← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 2099981] [NEW] [ovn] Adding an IPv6 subnet to public network breaks all FIPs

 

Public bug reported:

Description
===========

In case operator adds an IPv6 subnet to existing public shared network,
it cases outage for all Floating IPs created from such public network.

This happens because Neutron does add a new subnet to all Logical
Routers, causing external gateway port to be re-created. So UUID of the
external gateway port in OVN is changed.

With that it does update port UUIDs for NAT rules, while SRC-DST NAT
rules are binded to a specific LRP. So this causing FIPs to stop
functioning on external gateway update.


How to reproduce
================

Assuming you already have a network `public`:
```
root@aio1-utility-container-6c7d2d44:/# openstack subnet list --network public
+--------------------------------------+---------------+--------------------------------------+-----------------+
| ID                                   | Name          | Network                              | Subnet          |
+--------------------------------------+---------------+--------------------------------------+-----------------+
| 91f6448f-0927-4ef6-973b-203b996b960f | public-subnet | 06e11d8a-ff27-4b31-9b36-88419312ce20 | 172.29.248.0/22 |
+--------------------------------------+---------------+--------------------------------------+-----------------+
root@aio1-utility-container-6c7d2d44:/# 
```

Let's create a tenant network, server and a floating IP:
https://paste.openstack.org/show/biQnf6tadZHudz1I9yCa/


As a result we do have a valid resources in OVN NB:
```
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl get Logical_Router neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6 ports
[5115a2f4-1d99-4432-a5bc-edcee3ee4eb2, 88335cad-ce5c-4459-be27-9611c505b9ea]

root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl list Logical_Router_Port 88335cad-ce5c-4459-be27-9611c505b9ea
_uuid               : 88335cad-ce5c-4459-be27-9611c505b9ea
dhcp_relay          : []
enabled             : []
external_ids        : {"neutron:is_ext_gw"=True, "neutron:network_name"=neutron-06e11d8a-ff27-4b31-9b36-88419312ce20, "neutron:revision_number"="1", "neutron:router_name"="4172f3cb-743d-43b9-8e7d-4c1d3cd798a6", "neutron:subnet_ids"="91f6448f-0927-4ef6-973b-203b996b960f"}
gateway_chassis     : [8b135f0b-b1de-4b93-baff-edd617b13cca]
ha_chassis_group    : []
ipv6_prefix         : []
ipv6_ra_configs     : {}
mac                 : "fa:16:3e:be:c1:c7"
name                : lrp-6cfda7d8-8086-49f9-ada1-caedb2575c8a
networks            : ["172.29.249.118/22"]
options             : {gateway_mtu="1442", reside-on-redirect-chassis="true"}
peer                : []
status              : {hosting-chassis="11af7f06-3288-416e-a5c9-ad92bfd3388f"}
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl lr-nat-list neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6
TYPE             GATEWAY_PORT          MATCH                 EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP          EXTERNAL_MAC         LOGICAL_PORT
dnat_and_snat    lrp-6cfda7d8-8086-                          172.29.249.141                      192.168.10.69
snat                                                         172.29.249.118                      192.168.10.0/24
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
```

Now let's add an IPv6 subnet to our public network:

```
root@aio1-utility-container-6c7d2d44:/# openstack subnet pool list
+--------------------------------------+------------------+--------------------+
| ID                                   | Name             | Prefixes           |
+--------------------------------------+------------------+--------------------+
| 9a7e6a4b-b04c-4c52-92a5-9a07e9bd17a0 | ipv6_tenant_pool | 2a06:2982:300::/40 |
+--------------------------------------+------------------+--------------------+
root@aio1-utility-container-6c7d2d44:/# openstack subnet create --subnet-pool ipv6_tenant_pool --subnet-range 2a06:2982:300:1::/64 --gateway 2a06:2982:300:1:: --dhcp --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --network public --dns-nameserver 2001:4860:4860::8888 --dns-nameserver 2606:4700:4700::1111 public-v6-subnet
+----------------------+--------------------------------------------------------+
| Field                | Value                                                  |
+----------------------+--------------------------------------------------------+
| allocation_pools     | 2a06:2982:300:1::1-2a06:2982:300:1:ffff:ffff:ffff:ffff |
| cidr                 | 2a06:2982:300:1::/64                                   |
| created_at           | 2025-02-25T09:59:14Z                                   |
| description          |                                                        |
| dns_nameservers      | 2001:4860:4860::8888, 2606:4700:4700::1111             |
| dns_publish_fixed_ip | None                                                   |
| enable_dhcp          | True                                                   |
| gateway_ip           | 2a06:2982:300:1::                                      |
| host_routes          |                                                        |
| id                   | c9f4d361-4057-476a-b708-7337cf5e477e                   |
| ip_version           | 6                                                      |
| ipv6_address_mode    | slaac                                                  |
| ipv6_ra_mode         | slaac                                                  |
| name                 | public-v6-subnet                                       |
| network_id           | 06e11d8a-ff27-4b31-9b36-88419312ce20                   |
| project_id           | 232b649f2e1541dda247930e7672764e                       |
| revision_number      | 0                                                      |
| router:external      | True                                                   |
| segment_id           | None                                                   |
| service_types        |                                                        |
| subnetpool_id        | 9a7e6a4b-b04c-4c52-92a5-9a07e9bd17a0                   |
| tags                 |                                                        |
| updated_at           | 2025-02-25T09:59:14Z                                   |
+----------------------+--------------------------------------------------------+
root@aio1-utility-container-6c7d2d44:/#
```

With this, we get a broken behaviour for previously existing IPv4 FIPs in public network:
```
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl lr-nat-list neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6
TYPE             GATEWAY_PORT          MATCH                 EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP          EXTERNAL_MAC         LOGICAL_PORT
dnat_and_snat                                                172.29.249.141                      192.168.10.69
snat                                                         172.29.249.118                      192.168.10.0/24
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# 
```


You can also see that UUID of the external lrp is changed now - there's no more `88335cad-ce5c-4459-be27-9611c505b9ea`, it is now `13884c93-4e19-4707-9173-0cfce02fce4c`:
```
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl get Logical_Router neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6 ports
[13884c93-4e19-4707-9173-0cfce02fce4c, 5115a2f4-1d99-4432-a5bc-edcee3ee4eb2]
root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl list Logical_Router_Port 13884c93-4e19-4707-9173-0cfce02fce4c
_uuid               : 13884c93-4e19-4707-9173-0cfce02fce4c
dhcp_relay          : []
enabled             : []
external_ids        : {"neutron:is_ext_gw"=True, "neutron:network_name"=neutron-06e11d8a-ff27-4b31-9b36-88419312ce20, "neutron:revision_number"="6", "neutron:router_name"="4172f3cb-743d-43b9-8e7d-4c1d3cd798a6", "neutron:subnet_ids"="91f6448f-0927-4ef6-973b-203b996b960f c9f4d361-4057-476a-b708-7337cf5e477e"}
gateway_chassis     : [8b135f0b-b1de-4b93-baff-edd617b13cca]
ha_chassis_group    : []
ipv6_prefix         : []
ipv6_ra_configs     : {address_mode=slaac, mtu="1500", send_periodic="false"}
mac                 : "fa:16:3e:be:c1:c7"
name                : lrp-6cfda7d8-8086-49f9-ada1-caedb2575c8a
networks            : ["172.29.249.118/22", "2a06:2982:300:1:f816:3eff:febe:c1c7/64"]
options             : {gateway_mtu="1442", reside-on-redirect-chassis="true"}
peer                : []
status              : {hosting-chassis="11af7f06-3288-416e-a5c9-ad92bfd3388f"}
root@aio1-neutron-ovn-northd-container-a20c0cfc:/# 
```

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2099981

Title:
  [ovn] Adding an IPv6 subnet to public network breaks all FIPs

Status in neutron:
  New

Bug description:
  Description
  ===========

  In case operator adds an IPv6 subnet to existing public shared
  network, it cases outage for all Floating IPs created from such public
  network.

  This happens because Neutron does add a new subnet to all Logical
  Routers, causing external gateway port to be re-created. So UUID of
  the external gateway port in OVN is changed.

  With that it does update port UUIDs for NAT rules, while SRC-DST NAT
  rules are binded to a specific LRP. So this causing FIPs to stop
  functioning on external gateway update.

  
  How to reproduce
  ================

  Assuming you already have a network `public`:
  ```
  root@aio1-utility-container-6c7d2d44:/# openstack subnet list --network public
  +--------------------------------------+---------------+--------------------------------------+-----------------+
  | ID                                   | Name          | Network                              | Subnet          |
  +--------------------------------------+---------------+--------------------------------------+-----------------+
  | 91f6448f-0927-4ef6-973b-203b996b960f | public-subnet | 06e11d8a-ff27-4b31-9b36-88419312ce20 | 172.29.248.0/22 |
  +--------------------------------------+---------------+--------------------------------------+-----------------+
  root@aio1-utility-container-6c7d2d44:/# 
  ```

  Let's create a tenant network, server and a floating IP:
  https://paste.openstack.org/show/biQnf6tadZHudz1I9yCa/

  
  As a result we do have a valid resources in OVN NB:
  ```
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl get Logical_Router neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6 ports
  [5115a2f4-1d99-4432-a5bc-edcee3ee4eb2, 88335cad-ce5c-4459-be27-9611c505b9ea]

  root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl list Logical_Router_Port 88335cad-ce5c-4459-be27-9611c505b9ea
  _uuid               : 88335cad-ce5c-4459-be27-9611c505b9ea
  dhcp_relay          : []
  enabled             : []
  external_ids        : {"neutron:is_ext_gw"=True, "neutron:network_name"=neutron-06e11d8a-ff27-4b31-9b36-88419312ce20, "neutron:revision_number"="1", "neutron:router_name"="4172f3cb-743d-43b9-8e7d-4c1d3cd798a6", "neutron:subnet_ids"="91f6448f-0927-4ef6-973b-203b996b960f"}
  gateway_chassis     : [8b135f0b-b1de-4b93-baff-edd617b13cca]
  ha_chassis_group    : []
  ipv6_prefix         : []
  ipv6_ra_configs     : {}
  mac                 : "fa:16:3e:be:c1:c7"
  name                : lrp-6cfda7d8-8086-49f9-ada1-caedb2575c8a
  networks            : ["172.29.249.118/22"]
  options             : {gateway_mtu="1442", reside-on-redirect-chassis="true"}
  peer                : []
  status              : {hosting-chassis="11af7f06-3288-416e-a5c9-ad92bfd3388f"}
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl lr-nat-list neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6
  TYPE             GATEWAY_PORT          MATCH                 EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP          EXTERNAL_MAC         LOGICAL_PORT
  dnat_and_snat    lrp-6cfda7d8-8086-                          172.29.249.141                      192.168.10.69
  snat                                                         172.29.249.118                      192.168.10.0/24
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
  ```

  Now let's add an IPv6 subnet to our public network:

  ```
  root@aio1-utility-container-6c7d2d44:/# openstack subnet pool list
  +--------------------------------------+------------------+--------------------+
  | ID                                   | Name             | Prefixes           |
  +--------------------------------------+------------------+--------------------+
  | 9a7e6a4b-b04c-4c52-92a5-9a07e9bd17a0 | ipv6_tenant_pool | 2a06:2982:300::/40 |
  +--------------------------------------+------------------+--------------------+
  root@aio1-utility-container-6c7d2d44:/# openstack subnet create --subnet-pool ipv6_tenant_pool --subnet-range 2a06:2982:300:1::/64 --gateway 2a06:2982:300:1:: --dhcp --ip-version 6 --ipv6-ra-mode slaac --ipv6-address-mode slaac --network public --dns-nameserver 2001:4860:4860::8888 --dns-nameserver 2606:4700:4700::1111 public-v6-subnet
  +----------------------+--------------------------------------------------------+
  | Field                | Value                                                  |
  +----------------------+--------------------------------------------------------+
  | allocation_pools     | 2a06:2982:300:1::1-2a06:2982:300:1:ffff:ffff:ffff:ffff |
  | cidr                 | 2a06:2982:300:1::/64                                   |
  | created_at           | 2025-02-25T09:59:14Z                                   |
  | description          |                                                        |
  | dns_nameservers      | 2001:4860:4860::8888, 2606:4700:4700::1111             |
  | dns_publish_fixed_ip | None                                                   |
  | enable_dhcp          | True                                                   |
  | gateway_ip           | 2a06:2982:300:1::                                      |
  | host_routes          |                                                        |
  | id                   | c9f4d361-4057-476a-b708-7337cf5e477e                   |
  | ip_version           | 6                                                      |
  | ipv6_address_mode    | slaac                                                  |
  | ipv6_ra_mode         | slaac                                                  |
  | name                 | public-v6-subnet                                       |
  | network_id           | 06e11d8a-ff27-4b31-9b36-88419312ce20                   |
  | project_id           | 232b649f2e1541dda247930e7672764e                       |
  | revision_number      | 0                                                      |
  | router:external      | True                                                   |
  | segment_id           | None                                                   |
  | service_types        |                                                        |
  | subnetpool_id        | 9a7e6a4b-b04c-4c52-92a5-9a07e9bd17a0                   |
  | tags                 |                                                        |
  | updated_at           | 2025-02-25T09:59:14Z                                   |
  +----------------------+--------------------------------------------------------+
  root@aio1-utility-container-6c7d2d44:/#
  ```

  With this, we get a broken behaviour for previously existing IPv4 FIPs in public network:
  ```
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl lr-nat-list neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6
  TYPE             GATEWAY_PORT          MATCH                 EXTERNAL_IP        EXTERNAL_PORT    LOGICAL_IP          EXTERNAL_MAC         LOGICAL_PORT
  dnat_and_snat                                                172.29.249.141                      192.168.10.69
  snat                                                         172.29.249.118                      192.168.10.0/24
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/# 
  ```

  
  You can also see that UUID of the external lrp is changed now - there's no more `88335cad-ce5c-4459-be27-9611c505b9ea`, it is now `13884c93-4e19-4707-9173-0cfce02fce4c`:
  ```
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl get Logical_Router neutron-4172f3cb-743d-43b9-8e7d-4c1d3cd798a6 ports
  [13884c93-4e19-4707-9173-0cfce02fce4c, 5115a2f4-1d99-4432-a5bc-edcee3ee4eb2]
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/#
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/# ovn-nbctl list Logical_Router_Port 13884c93-4e19-4707-9173-0cfce02fce4c
  _uuid               : 13884c93-4e19-4707-9173-0cfce02fce4c
  dhcp_relay          : []
  enabled             : []
  external_ids        : {"neutron:is_ext_gw"=True, "neutron:network_name"=neutron-06e11d8a-ff27-4b31-9b36-88419312ce20, "neutron:revision_number"="6", "neutron:router_name"="4172f3cb-743d-43b9-8e7d-4c1d3cd798a6", "neutron:subnet_ids"="91f6448f-0927-4ef6-973b-203b996b960f c9f4d361-4057-476a-b708-7337cf5e477e"}
  gateway_chassis     : [8b135f0b-b1de-4b93-baff-edd617b13cca]
  ha_chassis_group    : []
  ipv6_prefix         : []
  ipv6_ra_configs     : {address_mode=slaac, mtu="1500", send_periodic="false"}
  mac                 : "fa:16:3e:be:c1:c7"
  name                : lrp-6cfda7d8-8086-49f9-ada1-caedb2575c8a
  networks            : ["172.29.249.118/22", "2a06:2982:300:1:f816:3eff:febe:c1c7/64"]
  options             : {gateway_mtu="1442", reside-on-redirect-chassis="true"}
  peer                : []
  status              : {hosting-chassis="11af7f06-3288-416e-a5c9-ad92bfd3388f"}
  root@aio1-neutron-ovn-northd-container-a20c0cfc:/# 
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2099981/+subscriptions