← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #94449

[Bug 2077532] [NEW] Two floatingip_agent_gateways are created which are bound to the same agent.

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

During the processing of a router by the L3 agent in `agent_mode={dvr|dvr_snat}`, we create a floatingip_agent_gateway port.
The code in the L3-agent  [0] and the neutron-server [1] assumes that there should be only one such port per L3-agent per network. However, this is not the case, and the parallel creation of two routers in the same external network can result in the creation of two floatingip_agent_gateway ports attached to a single L3-agent. The code [1] in the Neutron server is evidently susceptible to a race condition.

- Firstly, only one port will actually be used, leading to the waste of an additional IP address in the external network.
- Secondly, depending on the order in which the neutron-server returns these ports, we may encounter an error in router processing by the L3-agent, with a following retry attempt.

I deployed DevStack from the master branch on a <b>single<b> node and
reproduced the issue:

[root@devstack0 ~]# openstack router create r1 --external-gateway public & openstack router create r2 --external-gateway public
[root@devstack0 ~]# openstack port list --network public --device-owner network:floatingip_agent_gateway
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| ID                                   | Name | MAC Address       | Fixed IP Addresses                                                         | Status |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| 309ed611-d4f1-4521-9b94-c202484cd1ac |      | fa:16:3e:80:29:fe | ip_address='10.20.30.41', subnet_id='dc701931-ad94-4558-b078-69288cbb2190' | ACTIVE |
| ef195ebc-2f1a-4ede-8900-89bd8092062e |      | fa:16:3e:f0:60:2c | ip_address='10.20.30.58', subnet_id='dc701931-ad94-4558-b078-69288cbb2190' | ACTIVE |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
[root@devstack0 ~]#


Relevant neutron-server logs:

Aug 21 05:07:55 devstack0 neutron-server[127666]: INFO neutron.db.l3_dvr_db [None req-9b817aa9-748d-4666-8979-814c55737443 None None] Floating IP Agent Gateway port for network dad747c6-c234-41e3-ae27-c9602b81fbd2 does not exist on host devstack0. Creating one.
Aug 21 05:07:55 devstack0 neutron-server[127665]: INFO neutron.db.l3_dvr_db [None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Agent Gateway port for network dad747c6-c234-41e3-ae27-c9602b81fbd2 does not exist on host devstack0. Creating one.
Aug 21 05:07:55 devstack0 neutron-server[127665]: DEBUG neutron.db.l3_dvr_db [None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Gateway port agent binding for network dad747c6-c234-41e3-ae27-c9602b81fbd2 already exists on host devstack0. Probably it was just created by other worker. {{(pid=127665) create_fip_agent_gw_port_if_not_exists /opt/stack/neutron/neutron/db/l3_dvr_db.py:1201}}
...
Aug 21 05:07:55 devstack0 neutron-server[127665]: DEBUG neutron.db.l3_dvr_db [None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Agent Gateway port {'id': 'ef195ebc-2f1a-4ede-8900-89bd8092062e', 'name': '', 'network_id': 'dad747c6-c234-41e3-ae27-c9602b81fbd2', 'tenant_id': '', 'mac_address': 'fa:16:3e:f0:60:2c', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'c54e1e96-0e13-4ee1-94a9-3c355a377589', 'device_owner': 'network:floatingip_agent_gateway', 'standard_attr_id': 96, 'fixed_ips': [{'subnet_id': 'dc701931-ad94-4558-b078-69288cbb2190', 'ip_address': '10.20.30.58'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'devstack0', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': False, 'datapath_type': 'system', 'bridge_name': 'br-int'}, 'port_security_enabled': False, 'tags': [], 'created_at': '2024-08-21T09:07:55Z', 'updated_at': '2024-08-21T09:07:55Z', 'revision_number': 1, 'project_id': ''} created for the destination host: devstack0 {{(pid=127665) create_fip_agent_gw_port_if_not_exists /opt/stack/neutron/neutron/db/l3_dvr_db.py:1227}}
Aug 21 05:07:55 devstack0 neutron-server[127666]: DEBUG neutron.db.l3_dvr_db [None req-9b817aa9-748d-4666-8979-814c55737443 None None] Floating IP Agent Gateway port {'id': '309ed611-d4f1-4521-9b94-c202484cd1ac', 'name': '', 'network_id': 'dad747c6-c234-41e3-ae27-c9602b81fbd2', 'tenant_id': '', 'mac_address': 'fa:16:3e:80:29:fe', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'c54e1e96-0e13-4ee1-94a9-3c355a377589', 'device_owner': 'network:floatingip_agent_gateway', 'standard_attr_id': 95, 'fixed_ips': [{'subnet_id': 'dc701931-ad94-4558-b078-69288cbb2190', 'ip_address': '10.20.30.41'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'devstack0', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': False, 'datapath_type': 'system', 'bridge_name': 'br-int'}, 'port_security_enabled': False, 'tags': [], 'created_at': '2024-08-21T09:07:55Z', 'updated_at': '2024-08-21T09:07:55Z', 'revision_number': 1, 'project_id': ''} created for the destination host: devstack0 {{(pid=127666) create_fip_agent_gw_port_if_not_exists /opt/stack/neutron/neutron/db/l3_dvr_db.py:1227}}


[0] https://opendev.org/openstack/neutron/src/commit/0807c94dc9843fff318c21d1f6f7b8838f948f5f/neutron/agent/l3/dvr_local_router.py#L835-L839
[1] https://opendev.org/openstack/neutron/src/commit/0807c94dc9843fff318c21d1f6f7b8838f948f5f/neutron/db/l3_dvr_db.py#L1193-L1222

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2077532

Title:
  Two floatingip_agent_gateways are created which are bound to the same
  agent.

Status in neutron:
  New

Bug description:
  During the processing of a router by the L3 agent in `agent_mode={dvr|dvr_snat}`, we create a floatingip_agent_gateway port.
  The code in the L3-agent  [0] and the neutron-server [1] assumes that there should be only one such port per L3-agent per network. However, this is not the case, and the parallel creation of two routers in the same external network can result in the creation of two floatingip_agent_gateway ports attached to a single L3-agent. The code [1] in the Neutron server is evidently susceptible to a race condition.

  - Firstly, only one port will actually be used, leading to the waste of an additional IP address in the external network.
  - Secondly, depending on the order in which the neutron-server returns these ports, we may encounter an error in router processing by the L3-agent, with a following retry attempt.

  I deployed DevStack from the master branch on a <b>single<b> node and
  reproduced the issue:

  [root@devstack0 ~]# openstack router create r1 --external-gateway public & openstack router create r2 --external-gateway public
  [root@devstack0 ~]# openstack port list --network public --device-owner network:floatingip_agent_gateway
  +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
  | ID                                   | Name | MAC Address       | Fixed IP Addresses                                                         | Status |
  +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
  | 309ed611-d4f1-4521-9b94-c202484cd1ac |      | fa:16:3e:80:29:fe | ip_address='10.20.30.41', subnet_id='dc701931-ad94-4558-b078-69288cbb2190' | ACTIVE |
  | ef195ebc-2f1a-4ede-8900-89bd8092062e |      | fa:16:3e:f0:60:2c | ip_address='10.20.30.58', subnet_id='dc701931-ad94-4558-b078-69288cbb2190' | ACTIVE |
  +--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
  [root@devstack0 ~]#

  
  Relevant neutron-server logs:

  Aug 21 05:07:55 devstack0 neutron-server[127666]: INFO neutron.db.l3_dvr_db [None req-9b817aa9-748d-4666-8979-814c55737443 None None] Floating IP Agent Gateway port for network dad747c6-c234-41e3-ae27-c9602b81fbd2 does not exist on host devstack0. Creating one.
  Aug 21 05:07:55 devstack0 neutron-server[127665]: INFO neutron.db.l3_dvr_db [None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Agent Gateway port for network dad747c6-c234-41e3-ae27-c9602b81fbd2 does not exist on host devstack0. Creating one.
  Aug 21 05:07:55 devstack0 neutron-server[127665]: DEBUG neutron.db.l3_dvr_db [None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Gateway port agent binding for network dad747c6-c234-41e3-ae27-c9602b81fbd2 already exists on host devstack0. Probably it was just created by other worker. {{(pid=127665) create_fip_agent_gw_port_if_not_exists /opt/stack/neutron/neutron/db/l3_dvr_db.py:1201}}
  ...
  Aug 21 05:07:55 devstack0 neutron-server[127665]: DEBUG neutron.db.l3_dvr_db [None req-00a167c6-0cb7-4dcf-82a7-cee1d9a0fe14 None None] Floating IP Agent Gateway port {'id': 'ef195ebc-2f1a-4ede-8900-89bd8092062e', 'name': '', 'network_id': 'dad747c6-c234-41e3-ae27-c9602b81fbd2', 'tenant_id': '', 'mac_address': 'fa:16:3e:f0:60:2c', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'c54e1e96-0e13-4ee1-94a9-3c355a377589', 'device_owner': 'network:floatingip_agent_gateway', 'standard_attr_id': 96, 'fixed_ips': [{'subnet_id': 'dc701931-ad94-4558-b078-69288cbb2190', 'ip_address': '10.20.30.58'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'devstack0', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': False, 'datapath_type': 'system', 'bridge_name': 'br-int'}, 'port_security_enabled': False, 'tags': [], 'created_at': '2024-08-21T09:07:55Z', 'updated_at': '2024-08-21T09:07:55Z', 'revision_number': 1, 'project_id': ''} created for the destination host: devstack0 {{(pid=127665) create_fip_agent_gw_port_if_not_exists /opt/stack/neutron/neutron/db/l3_dvr_db.py:1227}}
  Aug 21 05:07:55 devstack0 neutron-server[127666]: DEBUG neutron.db.l3_dvr_db [None req-9b817aa9-748d-4666-8979-814c55737443 None None] Floating IP Agent Gateway port {'id': '309ed611-d4f1-4521-9b94-c202484cd1ac', 'name': '', 'network_id': 'dad747c6-c234-41e3-ae27-c9602b81fbd2', 'tenant_id': '', 'mac_address': 'fa:16:3e:80:29:fe', 'admin_state_up': True, 'status': 'DOWN', 'device_id': 'c54e1e96-0e13-4ee1-94a9-3c355a377589', 'device_owner': 'network:floatingip_agent_gateway', 'standard_attr_id': 95, 'fixed_ips': [{'subnet_id': 'dc701931-ad94-4558-b078-69288cbb2190', 'ip_address': '10.20.30.41'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': [], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'devstack0', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': False, 'datapath_type': 'system', 'bridge_name': 'br-int'}, 'port_security_enabled': False, 'tags': [], 'created_at': '2024-08-21T09:07:55Z', 'updated_at': '2024-08-21T09:07:55Z', 'revision_number': 1, 'project_id': ''} created for the destination host: devstack0 {{(pid=127666) create_fip_agent_gw_port_if_not_exists /opt/stack/neutron/neutron/db/l3_dvr_db.py:1227}}

  
  [0] https://opendev.org/openstack/neutron/src/commit/0807c94dc9843fff318c21d1f6f7b8838f948f5f/neutron/agent/l3/dvr_local_router.py#L835-L839
  [1] https://opendev.org/openstack/neutron/src/commit/0807c94dc9843fff318c21d1f6f7b8838f948f5f/neutron/db/l3_dvr_db.py#L1193-L1222

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2077532/+subscriptions
  Thread Previous • Date Previous • • Thread Next