← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 2028651] [NEW] IPv6 VIPs broken with ML2/OVN

 

Public bug reported:

Originally reported in the Octavia launchpad:
https://bugs.launchpad.net/octavia/+bug/2028524

The commit https://review.opendev.org/c/openstack/neutron/+/882588
introduced a regression in Octavia

It adds a validate_port_binding_and_virtual_port function that raises an exception when a port:
- has non-empty binding:host_id
- has fixed_ips/subnets
- has VIRTUAL type (in ovn)


When we create a load balancer in Octavia (with an IPv6 VIP)

$ openstack loadbalancer create --vip-subnet ipv6-public-subnet --name lb1
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| admin_state_up      | True                                 |
| availability_zone   | None                                 |
| created_at          | 2023-07-25T07:11:25                  |
| description         |                                      |
| flavor_id           | None                                 |
| id                  | 75cf51d2-4576-4878-8bfe-ad55584a7d76 |
| listeners           |                                      |
| name                | lb1                                  |
| operating_status    | OFFLINE                              |
| pools               |                                      |
| project_id          | 86f57e2e56874381a0d586263fc8d900     |
| provider            | amphora                              |
| provisioning_status | PENDING_CREATE                       |
| updated_at          | None                                 |
| vip_address         | 2001:db8::b1                         |
| vip_network_id      | 2d16ac53-8438-435d-a787-e5ceb4b783be |
| vip_port_id         | 83e51017-8f02-4916-bcd2-ebe0475b1ce6 |
| vip_qos_policy_id   | None                                 |
| vip_subnet_id       | 813adce0-21de-44c9-958a-6967441b8623 |
| tags                |                                      |
| additional_vips     | []                                   |
+---------------------+--------------------------------------+


The VIP port contains:

$ openstack port show 83e51017-8f02-4916-bcd2-ebe0475b1ce6
+-------------------------+--------------------------------------------------------------------------------------------------------+
| Field                   | Value                                                                                                  |
+-------------------------+--------------------------------------------------------------------------------------------------------+
| admin_state_up          | DOWN                                                                                                   |
| allowed_address_pairs   |                                                                                                        |
| binding_host_id         | gthiemon-devstack                                                                                      |
| binding_profile         |                                                                                                        |
| binding_vif_details     |                                                                                                        |
| binding_vif_type        | unbound                                                                                                |
| binding_vnic_type       | normal                                                                                                 |
| created_at              | 2023-07-25T07:11:25Z                                                                                   |
| data_plane_status       | None                                                                                                   |
| description             |                                                                                                        |
| device_id               | lb-75cf51d2-4576-4878-8bfe-ad55584a7d76                                                                |
| device_owner            | Octavia                                                                                                |
| device_profile          | None                                                                                                   |
| dns_assignment          | fqdn='host-2001-db8--b1.openstackgate.local.', hostname='host-2001-db8--b1', ip_address='2001:db8::b1' |
| dns_domain              |                                                                                                        |
| dns_name                |                                                                                                        |
| extra_dhcp_opts         |                                                                                                        |
| fixed_ips               | ip_address='2001:db8::b1', subnet_id='813adce0-21de-44c9-958a-6967441b8623'                            |
| id                      | 83e51017-8f02-4916-bcd2-ebe0475b1ce6                                                                   |
| ip_allocation           | None                                                                                                   |
| mac_address             | fa:16:3e:c9:4f:7e                                                                                      |
| name                    | octavia-lb-75cf51d2-4576-4878-8bfe-ad55584a7d76                                                        |
| network_id              | 2d16ac53-8438-435d-a787-e5ceb4b783be                                                                   |
| numa_affinity_policy    | None                                                                                                   |
| port_security_enabled   | True                                                                                                   |
| project_id              | 86f57e2e56874381a0d586263fc8d900                                                                       |
| propagate_uplink_status | None                                                                                                   |
| qos_network_policy_id   | None                                                                                                   |
| qos_policy_id           | None                                                                                                   |
| resource_request        | None                                                                                                   |
| revision_number         | 10                                                                                                     |
| security_group_ids      | 7c8d8935-9445-4e74-a815-a24246af757a                                                                   |
| status                  | DOWN                                                                                                   |
| tags                    |                                                                                                        |
| trunk_details           | None                                                                                                   |
| updated_at              | 2023-07-25T07:12:14Z                                                                                   |
+-------------------------+--------------------------------------------------------------------------------------------------------+


The port is not bound and has a binding_host_id, has a fixed_ips with a subnet and there's another port that has an allowed_address_pair with the VIP port's allocated address (so the port is a virtual port in OVN)


Any updates of this port result in a BadRequest Exception:

Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow Traceback (most recent call last):
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/octavia/octavia/network/drivers/neutron/base.py", line 129, in _add_security_group_to_port
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow self.network_proxy.update_port(
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/network/v2/_proxy.py", line 2979, in update_port
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return self._update(_port.Port, port, if_revision=if_revision, **attrs)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/proxy.py", line 64, in check Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return method(self, expected, actual, *args, **kwargs)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/network/v2/_proxy.py", line 189, in _update
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return res.commit(self, base_path=base_path, if_revision=if_revision) Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1794, in commit
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return self._commit(
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1839, in _commit
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow self._translate_response(response, has_body=has_body)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1278, in _translate_response
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow exceptions.raise_from_response(response, error_message=error_message)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/exceptions.py", line 263, in raise_from_response
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow raise cls(
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: http://192.168.1.101:9696/networking/v2.0/ports/618567c4-78c7-4398-b889-b567f6fd6aeb, Bad port request: A virtual logical switch port cannot be bound to a host.


The goal of this validation function seems to raise an exception when the binding_host_id is not empty, but the PortBindingUpdateVirtualPortsEvent class sets the binding_host_id of virtual ports.
https://opendev.org/openstack/neutron/src/commit/58c8493ff9defbb4544803ec3fc0432c0685c592/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py#L532-L537


Interestingly, it's not 100% reproducible (around 90%) and it is not reproducible with IPv4 VIPs, with IPv4 ports, the binding_host_id is always empty.

I have a couple of questions:
- is the validation function correct? should the binding_host_id be empty for VIP ports?
- why is binding_host_id set for IPv6 VIPs but not for IPv4 VIPs?

I can provide more logs if needed

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2028651

Title:
  IPv6 VIPs broken with ML2/OVN

Status in neutron:
  New

Bug description:
  Originally reported in the Octavia launchpad:
  https://bugs.launchpad.net/octavia/+bug/2028524

  The commit https://review.opendev.org/c/openstack/neutron/+/882588
  introduced a regression in Octavia

  It adds a validate_port_binding_and_virtual_port function that raises an exception when a port:
  - has non-empty binding:host_id
  - has fixed_ips/subnets
  - has VIRTUAL type (in ovn)

  
  When we create a load balancer in Octavia (with an IPv6 VIP)

  $ openstack loadbalancer create --vip-subnet ipv6-public-subnet --name lb1
  +---------------------+--------------------------------------+
  | Field               | Value                                |
  +---------------------+--------------------------------------+
  | admin_state_up      | True                                 |
  | availability_zone   | None                                 |
  | created_at          | 2023-07-25T07:11:25                  |
  | description         |                                      |
  | flavor_id           | None                                 |
  | id                  | 75cf51d2-4576-4878-8bfe-ad55584a7d76 |
  | listeners           |                                      |
  | name                | lb1                                  |
  | operating_status    | OFFLINE                              |
  | pools               |                                      |
  | project_id          | 86f57e2e56874381a0d586263fc8d900     |
  | provider            | amphora                              |
  | provisioning_status | PENDING_CREATE                       |
  | updated_at          | None                                 |
  | vip_address         | 2001:db8::b1                         |
  | vip_network_id      | 2d16ac53-8438-435d-a787-e5ceb4b783be |
  | vip_port_id         | 83e51017-8f02-4916-bcd2-ebe0475b1ce6 |
  | vip_qos_policy_id   | None                                 |
  | vip_subnet_id       | 813adce0-21de-44c9-958a-6967441b8623 |
  | tags                |                                      |
  | additional_vips     | []                                   |
  +---------------------+--------------------------------------+

  
  The VIP port contains:

  $ openstack port show 83e51017-8f02-4916-bcd2-ebe0475b1ce6
  +-------------------------+--------------------------------------------------------------------------------------------------------+
  | Field                   | Value                                                                                                  |
  +-------------------------+--------------------------------------------------------------------------------------------------------+
  | admin_state_up          | DOWN                                                                                                   |
  | allowed_address_pairs   |                                                                                                        |
  | binding_host_id         | gthiemon-devstack                                                                                      |
  | binding_profile         |                                                                                                        |
  | binding_vif_details     |                                                                                                        |
  | binding_vif_type        | unbound                                                                                                |
  | binding_vnic_type       | normal                                                                                                 |
  | created_at              | 2023-07-25T07:11:25Z                                                                                   |
  | data_plane_status       | None                                                                                                   |
  | description             |                                                                                                        |
  | device_id               | lb-75cf51d2-4576-4878-8bfe-ad55584a7d76                                                                |
  | device_owner            | Octavia                                                                                                |
  | device_profile          | None                                                                                                   |
  | dns_assignment          | fqdn='host-2001-db8--b1.openstackgate.local.', hostname='host-2001-db8--b1', ip_address='2001:db8::b1' |
  | dns_domain              |                                                                                                        |
  | dns_name                |                                                                                                        |
  | extra_dhcp_opts         |                                                                                                        |
  | fixed_ips               | ip_address='2001:db8::b1', subnet_id='813adce0-21de-44c9-958a-6967441b8623'                            |
  | id                      | 83e51017-8f02-4916-bcd2-ebe0475b1ce6                                                                   |
  | ip_allocation           | None                                                                                                   |
  | mac_address             | fa:16:3e:c9:4f:7e                                                                                      |
  | name                    | octavia-lb-75cf51d2-4576-4878-8bfe-ad55584a7d76                                                        |
  | network_id              | 2d16ac53-8438-435d-a787-e5ceb4b783be                                                                   |
  | numa_affinity_policy    | None                                                                                                   |
  | port_security_enabled   | True                                                                                                   |
  | project_id              | 86f57e2e56874381a0d586263fc8d900                                                                       |
  | propagate_uplink_status | None                                                                                                   |
  | qos_network_policy_id   | None                                                                                                   |
  | qos_policy_id           | None                                                                                                   |
  | resource_request        | None                                                                                                   |
  | revision_number         | 10                                                                                                     |
  | security_group_ids      | 7c8d8935-9445-4e74-a815-a24246af757a                                                                   |
  | status                  | DOWN                                                                                                   |
  | tags                    |                                                                                                        |
  | trunk_details           | None                                                                                                   |
  | updated_at              | 2023-07-25T07:12:14Z                                                                                   |
  +-------------------------+--------------------------------------------------------------------------------------------------------+

  
  The port is not bound and has a binding_host_id, has a fixed_ips with a subnet and there's another port that has an allowed_address_pair with the VIP port's allocated address (so the port is a virtual port in OVN)

  
  Any updates of this port result in a BadRequest Exception:

  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow Traceback (most recent call last):
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/octavia/octavia/network/drivers/neutron/base.py", line 129, in _add_security_group_to_port
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow self.network_proxy.update_port(
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/network/v2/_proxy.py", line 2979, in update_port
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return self._update(_port.Port, port, if_revision=if_revision, **attrs)
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/proxy.py", line 64, in check Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return method(self, expected, actual, *args, **kwargs)
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/network/v2/_proxy.py", line 189, in _update
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return res.commit(self, base_path=base_path, if_revision=if_revision) Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1794, in commit
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return self._commit(
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1839, in _commit
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow self._translate_response(response, has_body=has_body)
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1278, in _translate_response
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow exceptions.raise_from_response(response, error_message=error_message)
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/exceptions.py", line 263, in raise_from_response
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow raise cls(
  Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: http://192.168.1.101:9696/networking/v2.0/ports/618567c4-78c7-4398-b889-b567f6fd6aeb, Bad port request: A virtual logical switch port cannot be bound to a host.

  
  The goal of this validation function seems to raise an exception when the binding_host_id is not empty, but the PortBindingUpdateVirtualPortsEvent class sets the binding_host_id of virtual ports.
  https://opendev.org/openstack/neutron/src/commit/58c8493ff9defbb4544803ec3fc0432c0685c592/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py#L532-L537

  
  Interestingly, it's not 100% reproducible (around 90%) and it is not reproducible with IPv4 VIPs, with IPv4 ports, the binding_host_id is always empty.

  I have a couple of questions:
  - is the validation function correct? should the binding_host_id be empty for VIP ports?
  - why is binding_host_id set for IPv6 VIPs but not for IPv4 VIPs?

  I can provide more logs if needed

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2028651/+subscriptions



Follow ups