yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95395
[Bug 2099981] [NEW] [ovn] Adding an IPv6 subnet to public network breaks all FIPs
Public bug reported:
Description
===========
In case operator adds an IPv6 subnet to existing public shared network,
it cases outage for all Floating IPs created from such public network.
This happens because Neutron does add a new subnet to all Logical
Routers, causing external gateway port to be re-created. So UUID of the
external gateway port in OVN is changed.
With that it does update port UUIDs for NAT rules, while SRC-DST NAT
rules are binded to a specific LRP. So this causing FIPs to stop
functioning on external gateway update.
How to reproduce
================
Assuming you already have a network `public`:
```
root@aio1-utility-container-6c7d2d44:/# openstack subnet list --network public
+--------------------------------------+---------------+--------------------------------------+-----------------+
| ID | Name | Network | Subnet |
+--------------------------------------+---------------+--------------------------------------+-----------------+
| 91f6448f-0927-4ef6-973b-203b996b960f | public-subnet | 06e11d8a-ff27-4b31-9b36-88419312ce20 | 172.29.248.0/22 |
+--------------------------------------+---------------+--------------------------------------+-----------------+
root@aio1-utility-container-6c7d2d44:/#
```
Let's create a tenant network, server and a floating IP:
https://paste.openstack.org/show/biQnf6tadZHudz1I9yCa/
As a result we do have a valid resources in OVN NB:
```
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl get Logical_Router neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6 ports
[5115a2f4-1d99-4432-a5bc-edcee3ee4eb2, 88335cad-ce5c-4459-be27-9611c505b9ea]
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl list Logical_Router_Port 88335cad-ce5c-4459-be27-9611c505b9ea
_uuid : 88335cad-ce5c-4459-be27-9611c505b9ea
dhcp_relay : []
enabled : []
external_ids : {"neutron:is_ext_gw"=True, "neutron:network_name"=neutron-06e11d8a-ff27-4b31-9b36-88419312ce20, "neutron:revision_number"="1", "neutron:router_name"="4172f3cb-743d-43b9-8e7d-4c1d3cd798a6", "neutron:subnet_ids"="91f6448f-0927-4ef6-973b-203b996b960f"}
gateway_chassis : [8b135f0b-b1de-4b93-baff-edd617b13cca]
ha_chassis_group : []
ipv6_prefix : []
ipv6_ra_configs : {}
mac : "fa:16:3e:be:c1:c7"
name : lrp-6cfda7d8-8086-49f9-ada1-caedb2575c8a
networks : ["172.29.249.118/22"]
options : {gateway_mtu="1442", reside-on-redirect-chassis="true"}
peer : []
status : {hosting-chassis="11af7f06-3288-416e-a5c9-ad92bfd3388f"}
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl lr-nat-list neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6
TYPE GATEWAY_PORT MATCH EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT
dnat_and_snat lrp-6cfda7d8-8086- 172.29.249.141 192.168.10.69
snat 172.29.249.118 192.168.10.0/24
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
```
Now let's add an IPv6 subnet to our public network:
```
root@aio1-utility-container-6c7d2d44:/# openstack subnet pool list
+--------------------------------------+------------------+--------------------+
| ID | Name | Prefixes |
+--------------------------------------+------------------+--------------------+
| 9a7e6a4b-b04c-4c52-92a5-9a07e9bd17a0 | ipv6_tenant_pool | 2a06:2982:300::/40 |
+--------------------------------------+------------------+--------------------+
root@aio1-utility-container-6c7d2d44:/# openstack subnet create --subnet-pool ipv6_tenant_pool --subnet-range 2a06:2982:300:1::/64 --gateway 2a06:2982:300:1:: --dhcp --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --network public --dns-nameserver 2001:4860:4860::8888 --dns-nameserver 2606:4700:4700::1111 public-v6-subnet
+----------------------+--------------------------------------------------------+
| Field | Value |
+----------------------+--------------------------------------------------------+
| allocation_pools | 2a06:2982:300:1::1-2a06:2982:300:1:ffff:ffff:ffff:ffff |
| cidr | 2a06:2982:300:1::/64 |
| created_at | 2025-02-25T09:59:14Z |
| description | |
| dns_nameservers | 2001:4860:4860::8888, 2606:4700:4700::1111 |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 2a06:2982:300:1:: |
| host_routes | |
| id | c9f4d361-4057-476a-b708-7337cf5e477e |
| ip_version | 6 |
| ipv6_address_mode | slaac |
| ipv6_ra_mode | slaac |
| name | public-v6-subnet |
| network_id | 06e11d8a-ff27-4b31-9b36-88419312ce20 |
| project_id | 232b649f2e1541dda247930e7672764e |
| revision_number | 0 |
| router:external | True |
| segment_id | None |
| service_types | |
| subnetpool_id | 9a7e6a4b-b04c-4c52-92a5-9a07e9bd17a0 |
| tags | |
| updated_at | 2025-02-25T09:59:14Z |
+----------------------+--------------------------------------------------------+
root@aio1-utility-container-6c7d2d44:/#
```
With this, we get a broken behaviour for previously existing IPv4 FIPs in public network:
```
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl lr-nat-list neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6
TYPE GATEWAY_PORT MATCH EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT
dnat_and_snat 172.29.249.141 192.168.10.69
snat 172.29.249.118 192.168.10.0/24
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
```
You can also see that UUID of the external lrp is changed now - there's no more `88335cad-ce5c-4459-be27-9611c505b9ea`, it is now `13884c93-4e19-4707-9173-0cfce02fce4c`:
```
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl get Logical_Router neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6 ports
[13884c93-4e19-4707-9173-0cfce02fce4c, 5115a2f4-1d99-4432-a5bc-edcee3ee4eb2]
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl list Logical_Router_Port 13884c93-4e19-4707-9173-0cfce02fce4c
_uuid : 13884c93-4e19-4707-9173-0cfce02fce4c
dhcp_relay : []
enabled : []
external_ids : {"neutron:is_ext_gw"=True, "neutron:network_name"=neutron-06e11d8a-ff27-4b31-9b36-88419312ce20, "neutron:revision_number"="6", "neutron:router_name"="4172f3cb-743d-43b9-8e7d-4c1d3cd798a6", "neutron:subnet_ids"="91f6448f-0927-4ef6-973b-203b996b960f c9f4d361-4057-476a-b708-7337cf5e477e"}
gateway_chassis : [8b135f0b-b1de-4b93-baff-edd617b13cca]
ha_chassis_group : []
ipv6_prefix : []
ipv6_ra_configs : {address_mode=slaac, mtu="1500", send_periodic="false"}
mac : "fa:16:3e:be:c1:c7"
name : lrp-6cfda7d8-8086-49f9-ada1-caedb2575c8a
networks : ["172.29.249.118/22", "2a06:2982:300:1:f816:3eff:febe:c1c7/64"]
options : {gateway_mtu="1442", reside-on-redirect-chassis="true"}
peer : []
status : {hosting-chassis="11af7f06-3288-416e-a5c9-ad92bfd3388f"}
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
```
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2099981
Title:
[ovn] Adding an IPv6 subnet to public network breaks all FIPs
Status in neutron:
New
Bug description:
Description
===========
In case operator adds an IPv6 subnet to existing public shared
network, it cases outage for all Floating IPs created from such public
network.
This happens because Neutron does add a new subnet to all Logical
Routers, causing external gateway port to be re-created. So UUID of
the external gateway port in OVN is changed.
With that it does update port UUIDs for NAT rules, while SRC-DST NAT
rules are binded to a specific LRP. So this causing FIPs to stop
functioning on external gateway update.
How to reproduce
================
Assuming you already have a network `public`:
```
root@aio1-utility-container-6c7d2d44:/# openstack subnet list --network public
+--------------------------------------+---------------+--------------------------------------+-----------------+
| ID | Name | Network | Subnet |
+--------------------------------------+---------------+--------------------------------------+-----------------+
| 91f6448f-0927-4ef6-973b-203b996b960f | public-subnet | 06e11d8a-ff27-4b31-9b36-88419312ce20 | 172.29.248.0/22 |
+--------------------------------------+---------------+--------------------------------------+-----------------+
root@aio1-utility-container-6c7d2d44:/#
```
Let's create a tenant network, server and a floating IP:
https://paste.openstack.org/show/biQnf6tadZHudz1I9yCa/
As a result we do have a valid resources in OVN NB:
```
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl get Logical_Router neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6 ports
[5115a2f4-1d99-4432-a5bc-edcee3ee4eb2, 88335cad-ce5c-4459-be27-9611c505b9ea]
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl list Logical_Router_Port 88335cad-ce5c-4459-be27-9611c505b9ea
_uuid : 88335cad-ce5c-4459-be27-9611c505b9ea
dhcp_relay : []
enabled : []
external_ids : {"neutron:is_ext_gw"=True, "neutron:network_name"=neutron-06e11d8a-ff27-4b31-9b36-88419312ce20, "neutron:revision_number"="1", "neutron:router_name"="4172f3cb-743d-43b9-8e7d-4c1d3cd798a6", "neutron:subnet_ids"="91f6448f-0927-4ef6-973b-203b996b960f"}
gateway_chassis : [8b135f0b-b1de-4b93-baff-edd617b13cca]
ha_chassis_group : []
ipv6_prefix : []
ipv6_ra_configs : {}
mac : "fa:16:3e:be:c1:c7"
name : lrp-6cfda7d8-8086-49f9-ada1-caedb2575c8a
networks : ["172.29.249.118/22"]
options : {gateway_mtu="1442", reside-on-redirect-chassis="true"}
peer : []
status : {hosting-chassis="11af7f06-3288-416e-a5c9-ad92bfd3388f"}
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl lr-nat-list neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6
TYPE GATEWAY_PORT MATCH EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT
dnat_and_snat lrp-6cfda7d8-8086- 172.29.249.141 192.168.10.69
snat 172.29.249.118 192.168.10.0/24
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
```
Now let's add an IPv6 subnet to our public network:
```
root@aio1-utility-container-6c7d2d44:/# openstack subnet pool list
+--------------------------------------+------------------+--------------------+
| ID | Name | Prefixes |
+--------------------------------------+------------------+--------------------+
| 9a7e6a4b-b04c-4c52-92a5-9a07e9bd17a0 | ipv6_tenant_pool | 2a06:2982:300::/40 |
+--------------------------------------+------------------+--------------------+
root@aio1-utility-container-6c7d2d44:/# openstack subnet create --subnet-pool ipv6_tenant_pool --subnet-range 2a06:2982:300:1::/64 --gateway 2a06:2982:300:1:: --dhcp --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --network public --dns-nameserver 2001:4860:4860::8888 --dns-nameserver 2606:4700:4700::1111 public-v6-subnet
+----------------------+--------------------------------------------------------+
| Field | Value |
+----------------------+--------------------------------------------------------+
| allocation_pools | 2a06:2982:300:1::1-2a06:2982:300:1:ffff:ffff:ffff:ffff |
| cidr | 2a06:2982:300:1::/64 |
| created_at | 2025-02-25T09:59:14Z |
| description | |
| dns_nameservers | 2001:4860:4860::8888, 2606:4700:4700::1111 |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 2a06:2982:300:1:: |
| host_routes | |
| id | c9f4d361-4057-476a-b708-7337cf5e477e |
| ip_version | 6 |
| ipv6_address_mode | slaac |
| ipv6_ra_mode | slaac |
| name | public-v6-subnet |
| network_id | 06e11d8a-ff27-4b31-9b36-88419312ce20 |
| project_id | 232b649f2e1541dda247930e7672764e |
| revision_number | 0 |
| router:external | True |
| segment_id | None |
| service_types | |
| subnetpool_id | 9a7e6a4b-b04c-4c52-92a5-9a07e9bd17a0 |
| tags | |
| updated_at | 2025-02-25T09:59:14Z |
+----------------------+--------------------------------------------------------+
root@aio1-utility-container-6c7d2d44:/#
```
With this, we get a broken behaviour for previously existing IPv4 FIPs in public network:
```
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl lr-nat-list neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6
TYPE GATEWAY_PORT MATCH EXTERNAL_IP EXTERNAL_PORT LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT
dnat_and_snat 172.29.249.141 192.168.10.69
snat 172.29.249.118 192.168.10.0/24
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
```
You can also see that UUID of the external lrp is changed now - there's no more `88335cad-ce5c-4459-be27-9611c505b9ea`, it is now `13884c93-4e19-4707-9173-0cfce02fce4c`:
```
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl get Logical_Router neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6 ports
[13884c93-4e19-4707-9173-0cfce02fce4c, 5115a2f4-1d99-4432-a5bc-edcee3ee4eb2]
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl list Logical_Router_Port 13884c93-4e19-4707-9173-0cfce02fce4c
_uuid : 13884c93-4e19-4707-9173-0cfce02fce4c
dhcp_relay : []
enabled : []
external_ids : {"neutron:is_ext_gw"=True, "neutron:network_name"=neutron-06e11d8a-ff27-4b31-9b36-88419312ce20, "neutron:revision_number"="6", "neutron:router_name"="4172f3cb-743d-43b9-8e7d-4c1d3cd798a6", "neutron:subnet_ids"="91f6448f-0927-4ef6-973b-203b996b960f c9f4d361-4057-476a-b708-7337cf5e477e"}
gateway_chassis : [8b135f0b-b1de-4b93-baff-edd617b13cca]
ha_chassis_group : []
ipv6_prefix : []
ipv6_ra_configs : {address_mode=slaac, mtu="1500", send_periodic="false"}
mac : "fa:16:3e:be:c1:c7"
name : lrp-6cfda7d8-8086-49f9-ada1-caedb2575c8a
networks : ["172.29.249.118/22", "2a06:2982:300:1:f816:3eff:febe:c1c7/64"]
options : {gateway_mtu="1442", reside-on-redirect-chassis="true"}
peer : []
status : {hosting-chassis="11af7f06-3288-416e-a5c9-ad92bfd3388f"}
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
```
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2099981/+subscriptions
