yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #92678
[Bug 2028651] [NEW] IPv6 VIPs broken with ML2/OVN
Public bug reported:
Originally reported in the Octavia launchpad:
https://bugs.launchpad.net/octavia/+bug/2028524
The commit https://review.opendev.org/c/openstack/neutron/+/882588
introduced a regression in Octavia
It adds a validate_port_binding_and_virtual_port function that raises an exception when a port:
- has non-empty binding:host_id
- has fixed_ips/subnets
- has VIRTUAL type (in ovn)
When we create a load balancer in Octavia (with an IPv6 VIP)
$ openstack loadbalancer create --vip-subnet ipv6-public-subnet --name lb1
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| admin_state_up | True |
| availability_zone | None |
| created_at | 2023-07-25T07:11:25 |
| description | |
| flavor_id | None |
| id | 75cf51d2-4576-4878-8bfe-ad55584a7d76 |
| listeners | |
| name | lb1 |
| operating_status | OFFLINE |
| pools | |
| project_id | 86f57e2e56874381a0d586263fc8d900 |
| provider | amphora |
| provisioning_status | PENDING_CREATE |
| updated_at | None |
| vip_address | 2001:db8::b1 |
| vip_network_id | 2d16ac53-8438-435d-a787-e5ceb4b783be |
| vip_port_id | 83e51017-8f02-4916-bcd2-ebe0475b1ce6 |
| vip_qos_policy_id | None |
| vip_subnet_id | 813adce0-21de-44c9-958a-6967441b8623 |
| tags | |
| additional_vips | [] |
+---------------------+--------------------------------------+
The VIP port contains:
$ openstack port show 83e51017-8f02-4916-bcd2-ebe0475b1ce6
+-------------------------+--------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------------------------------------------------------------------------+
| admin_state_up | DOWN |
| allowed_address_pairs | |
| binding_host_id | gthiemon-devstack |
| binding_profile | |
| binding_vif_details | |
| binding_vif_type | unbound |
| binding_vnic_type | normal |
| created_at | 2023-07-25T07:11:25Z |
| data_plane_status | None |
| description | |
| device_id | lb-75cf51d2-4576-4878-8bfe-ad55584a7d76 |
| device_owner | Octavia |
| device_profile | None |
| dns_assignment | fqdn='host-2001-db8--b1.openstackgate.local.', hostname='host-2001-db8--b1', ip_address='2001:db8::b1' |
| dns_domain | |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | ip_address='2001:db8::b1', subnet_id='813adce0-21de-44c9-958a-6967441b8623' |
| id | 83e51017-8f02-4916-bcd2-ebe0475b1ce6 |
| ip_allocation | None |
| mac_address | fa:16:3e:c9:4f:7e |
| name | octavia-lb-75cf51d2-4576-4878-8bfe-ad55584a7d76 |
| network_id | 2d16ac53-8438-435d-a787-e5ceb4b783be |
| numa_affinity_policy | None |
| port_security_enabled | True |
| project_id | 86f57e2e56874381a0d586263fc8d900 |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 10 |
| security_group_ids | 7c8d8935-9445-4e74-a815-a24246af757a |
| status | DOWN |
| tags | |
| trunk_details | None |
| updated_at | 2023-07-25T07:12:14Z |
+-------------------------+--------------------------------------------------------------------------------------------------------+
The port is not bound and has a binding_host_id, has a fixed_ips with a subnet and there's another port that has an allowed_address_pair with the VIP port's allocated address (so the port is a virtual port in OVN)
Any updates of this port result in a BadRequest Exception:
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow Traceback (most recent call last):
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/octavia/octavia/network/drivers/neutron/base.py", line 129, in _add_security_group_to_port
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow self.network_proxy.update_port(
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/network/v2/_proxy.py", line 2979, in update_port
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return self._update(_port.Port, port, if_revision=if_revision, **attrs)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/proxy.py", line 64, in check Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return method(self, expected, actual, *args, **kwargs)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/network/v2/_proxy.py", line 189, in _update
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return res.commit(self, base_path=base_path, if_revision=if_revision) Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1794, in commit
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return self._commit(
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1839, in _commit
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow self._translate_response(response, has_body=has_body)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1278, in _translate_response
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow exceptions.raise_from_response(response, error_message=error_message)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/exceptions.py", line 263, in raise_from_response
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow raise cls(
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: http://192.168.1.101:9696/networking/v2.0/ports/618567c4-78c7-4398-b889-b567f6fd6aeb, Bad port request: A virtual logical switch port cannot be bound to a host.
The goal of this validation function seems to raise an exception when the binding_host_id is not empty, but the PortBindingUpdateVirtualPortsEvent class sets the binding_host_id of virtual ports.
https://opendev.org/openstack/neutron/src/commit/58c8493ff9defbb4544803ec3fc0432c0685c592/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py#L532-L537
Interestingly, it's not 100% reproducible (around 90%) and it is not reproducible with IPv4 VIPs, with IPv4 ports, the binding_host_id is always empty.
I have a couple of questions:
- is the validation function correct? should the binding_host_id be empty for VIP ports?
- why is binding_host_id set for IPv6 VIPs but not for IPv4 VIPs?
I can provide more logs if needed
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2028651
Title:
IPv6 VIPs broken with ML2/OVN
Status in neutron:
New
Bug description:
Originally reported in the Octavia launchpad:
https://bugs.launchpad.net/octavia/+bug/2028524
The commit https://review.opendev.org/c/openstack/neutron/+/882588
introduced a regression in Octavia
It adds a validate_port_binding_and_virtual_port function that raises an exception when a port:
- has non-empty binding:host_id
- has fixed_ips/subnets
- has VIRTUAL type (in ovn)
When we create a load balancer in Octavia (with an IPv6 VIP)
$ openstack loadbalancer create --vip-subnet ipv6-public-subnet --name lb1
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| admin_state_up | True |
| availability_zone | None |
| created_at | 2023-07-25T07:11:25 |
| description | |
| flavor_id | None |
| id | 75cf51d2-4576-4878-8bfe-ad55584a7d76 |
| listeners | |
| name | lb1 |
| operating_status | OFFLINE |
| pools | |
| project_id | 86f57e2e56874381a0d586263fc8d900 |
| provider | amphora |
| provisioning_status | PENDING_CREATE |
| updated_at | None |
| vip_address | 2001:db8::b1 |
| vip_network_id | 2d16ac53-8438-435d-a787-e5ceb4b783be |
| vip_port_id | 83e51017-8f02-4916-bcd2-ebe0475b1ce6 |
| vip_qos_policy_id | None |
| vip_subnet_id | 813adce0-21de-44c9-958a-6967441b8623 |
| tags | |
| additional_vips | [] |
+---------------------+--------------------------------------+
The VIP port contains:
$ openstack port show 83e51017-8f02-4916-bcd2-ebe0475b1ce6
+-------------------------+--------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+--------------------------------------------------------------------------------------------------------+
| admin_state_up | DOWN |
| allowed_address_pairs | |
| binding_host_id | gthiemon-devstack |
| binding_profile | |
| binding_vif_details | |
| binding_vif_type | unbound |
| binding_vnic_type | normal |
| created_at | 2023-07-25T07:11:25Z |
| data_plane_status | None |
| description | |
| device_id | lb-75cf51d2-4576-4878-8bfe-ad55584a7d76 |
| device_owner | Octavia |
| device_profile | None |
| dns_assignment | fqdn='host-2001-db8--b1.openstackgate.local.', hostname='host-2001-db8--b1', ip_address='2001:db8::b1' |
| dns_domain | |
| dns_name | |
| extra_dhcp_opts | |
| fixed_ips | ip_address='2001:db8::b1', subnet_id='813adce0-21de-44c9-958a-6967441b8623' |
| id | 83e51017-8f02-4916-bcd2-ebe0475b1ce6 |
| ip_allocation | None |
| mac_address | fa:16:3e:c9:4f:7e |
| name | octavia-lb-75cf51d2-4576-4878-8bfe-ad55584a7d76 |
| network_id | 2d16ac53-8438-435d-a787-e5ceb4b783be |
| numa_affinity_policy | None |
| port_security_enabled | True |
| project_id | 86f57e2e56874381a0d586263fc8d900 |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 10 |
| security_group_ids | 7c8d8935-9445-4e74-a815-a24246af757a |
| status | DOWN |
| tags | |
| trunk_details | None |
| updated_at | 2023-07-25T07:12:14Z |
+-------------------------+--------------------------------------------------------------------------------------------------------+
The port is not bound and has a binding_host_id, has a fixed_ips with a subnet and there's another port that has an allowed_address_pair with the VIP port's allocated address (so the port is a virtual port in OVN)
Any updates of this port result in a BadRequest Exception:
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow Traceback (most recent call last):
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/octavia/octavia/network/drivers/neutron/base.py", line 129, in _add_security_group_to_port
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow self.network_proxy.update_port(
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/network/v2/_proxy.py", line 2979, in update_port
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return self._update(_port.Port, port, if_revision=if_revision, **attrs)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/proxy.py", line 64, in check Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return method(self, expected, actual, *args, **kwargs)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/network/v2/_proxy.py", line 189, in _update
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return res.commit(self, base_path=base_path, if_revision=if_revision) Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1794, in commit
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow return self._commit(
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1839, in _commit
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow self._translate_response(response, has_body=has_body)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/resource.py", line 1278, in _translate_response
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow exceptions.raise_from_response(response, error_message=error_message)
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow File "/opt/stack/openstacksdk/openstack/exceptions.py", line 263, in raise_from_response
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow raise cls(
Jul 24 03:08:07 gthiemon-devstack octavia-worker[97901]: ERROR octavia.common.base_taskflow openstack.exceptions.BadRequestException: BadRequestException: 400: Client Error for url: http://192.168.1.101:9696/networking/v2.0/ports/618567c4-78c7-4398-b889-b567f6fd6aeb, Bad port request: A virtual logical switch port cannot be bound to a host.
The goal of this validation function seems to raise an exception when the binding_host_id is not empty, but the PortBindingUpdateVirtualPortsEvent class sets the binding_host_id of virtual ports.
https://opendev.org/openstack/neutron/src/commit/58c8493ff9defbb4544803ec3fc0432c0685c592/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovsdb_monitor.py#L532-L537
Interestingly, it's not 100% reproducible (around 90%) and it is not reproducible with IPv4 VIPs, with IPv4 ports, the binding_host_id is always empty.
I have a couple of questions:
- is the validation function correct? should the binding_host_id be empty for VIP ports?
- why is binding_host_id set for IPv6 VIPs but not for IPv4 VIPs?
I can provide more logs if needed
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2028651/+subscriptions
Follow ups
