yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1866336] [NEW] Binding of floating ip agent gateway port and agent_id isn't removed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1866336@xxxxxxxxxxxxxxxxxx>
Date: Fri, 06 Mar 2020 12:00:56 -0000
Reply-to: Bug 1866336 <1866336@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

In patch https://review.opendev.org/#/c/702547/ new db level lock was introduced to ensure that there is always max. 1 FIP gateway port for network on the host.
But unfortunately I missed to clean entry in this new table when such gateway port is removed. That causes issues with DVR routers when gateway is plug to the router.

Steps to reproduce issue:

1. Create dvr router
2. Attach some network with vm to the router,
3. Attach external gateway network to the router,
4. It will create fip- namespace on compute,
5. remove this router, fip- namespace will be cleaned and agent's gateway port will be removed from neutron db,
6. repeat steps 1-5 - it will now fail with error like:

Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'get'
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: Traceback (most recent call last):
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     res = self.dispatcher.dispatch(message)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return self._do_dispatch(endpoint, method, ctxt, args)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     result = func(ctxt, **new_args)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     setattr(e, '_RETRY_EXCEEDED', True)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self.force_reraise()
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     six.reraise(self.type_, self.value, self.tb)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     raise value
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return f(*args, **kwargs)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 154, in wrapper
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     ectxt.value = e.inner_exc
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self.force_reraise()
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     six.reraise(self.type_, self.value, self.tb)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     raise value
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 142, in wrapper
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return f(*args, **kwargs)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     LOG.debug("Retry wrapper got retriable exception: %s", e)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self.force_reraise()
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     six.reraise(self.type_, self.value, self.tb)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     raise value
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return f(*dup_args, **dup_kwargs)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 319, in get_agent_gateway_port
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     admin_ctx, network_id, host)
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 1078, in create_fip_agent_gw_port_if_not_exists
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self._populate_mtu_and_subnets_for_ports(context, [agent_port])
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1762, in _populate_mtu_and_subnets_for_ports
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     for p in self._each_port_having_fixed_ips(ports)]
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1761, in <listcomp>
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     network_ids = [p['network_id']
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1709, in _each_port_having_fixed_ips
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     fixed_ips = port.get('fixed_ips', [])
Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: AttributeError: 'NoneType' object has no attribute 'get'


To solve this issue You need to manually remove corresponding row from "dvr_fip_gateway_port_network" db table.

** Affects: neutron
     Importance: High
     Assignee: Slawek Kaplonski (slaweq)
         Status: Confirmed


** Tags: l3-dvr-backlog

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1866336

Title:
  Binding of floating ip agent gateway port and agent_id isn't removed

Status in neutron:
  Confirmed

Bug description:
  In patch https://review.opendev.org/#/c/702547/ new db level lock was introduced to ensure that there is always max. 1 FIP gateway port for network on the host.
  But unfortunately I missed to clean entry in this new table when such gateway port is removed. That causes issues with DVR routers when gateway is plug to the router.

  Steps to reproduce issue:

  1. Create dvr router
  2. Attach some network with vm to the router,
  3. Attach external gateway network to the router,
  4. It will create fip- namespace on compute,
  5. remove this router, fip- namespace will be cleaned and agent's gateway port will be removed from neutron db,
  6. repeat steps 1-5 - it will now fail with error like:

  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'get'
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: Traceback (most recent call last):
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     res = self.dispatcher.dispatch(message)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return self._do_dispatch(endpoint, method, ctxt, args)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     result = func(ctxt, **new_args)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     setattr(e, '_RETRY_EXCEEDED', True)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self.force_reraise()
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     six.reraise(self.type_, self.value, self.tb)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     raise value
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return f(*args, **kwargs)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 154, in wrapper
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     ectxt.value = e.inner_exc
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self.force_reraise()
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     six.reraise(self.type_, self.value, self.tb)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     raise value
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 142, in wrapper
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return f(*args, **kwargs)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     LOG.debug("Retry wrapper got retriable exception: %s", e)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self.force_reraise()
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     six.reraise(self.type_, self.value, self.tb)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     raise value
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return f(*dup_args, **dup_kwargs)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 319, in get_agent_gateway_port
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     admin_ctx, network_id, host)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 1078, in create_fip_agent_gw_port_if_not_exists
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self._populate_mtu_and_subnets_for_ports(context, [agent_port])
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1762, in _populate_mtu_and_subnets_for_ports
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     for p in self._each_port_having_fixed_ips(ports)]
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1761, in <listcomp>
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     network_ids = [p['network_id']
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1709, in _each_port_having_fixed_ips
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     fixed_ips = port.get('fixed_ips', [])
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: AttributeError: 'NoneType' object has no attribute 'get'

  
  To solve this issue You need to manually remove corresponding row from "dvr_fip_gateway_port_network" db table.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1866336/+subscriptions
Follow ups

[Bug 1866336] Re: Binding of floating ip agent gateway port and agent_id isn't removed
From: OpenStack Infra, 2020-03-18