yahoo-eng-team team mailing list archive

[OpenStack Infra <1866336@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/711623
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2baeae75199405d46927c073daf772b2a4144294
Submitter: Zuul
Branch:    master

commit 2baeae75199405d46927c073daf772b2a4144294
Author: Slawek Kaplonski <skaplons@xxxxxxxxxx>
Date:   Mon Mar 2 13:56:21 2020 +0100

    Clean dvr fip gateway entry when fip agent gw port is deleted
    
    In [1] there was introduced new db table which stored information about
    which DVR L3 agent has got already floating ip gateway port. It was to
    avoid race conditions and ensure that there is always only one such port
    per network and per agent (host).
    Unfortunately in [1] there was no added removal of correct record from
    this db table so it was causing problems when such port had to be
    recreated after it was already on the host and was deleted.
    
    This patch adds removal of such entry from db when needed.
    
    Closes-Bug: #1866336
    
    [1] https://review.opendev.org/#/c/702547/
    
    Change-Id: I56efd1b9f09c0449ce531a185fcf4db353f99fe1


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1866336

Title:
  Binding of floating ip agent gateway port and agent_id isn't removed

Status in neutron:
  Fix Released

Bug description:
  In patch https://review.opendev.org/#/c/702547/ new db level lock was introduced to ensure that there is always max. 1 FIP gateway port for network on the host.
  But unfortunately I missed to clean entry in this new table when such gateway port is removed. That causes issues with DVR routers when gateway is plug to the router.

  Steps to reproduce issue:

  1. Create dvr router
  2. Attach some network with vm to the router,
  3. Attach external gateway network to the router,
  4. It will create fip- namespace on compute,
  5. remove this router, fip- namespace will be cleaned and agent's gateway port will be removed from neutron db,
  6. repeat steps 1-5 - it will now fail with error like:

  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: ERROR neutron.agent.l3.router_info [-] 'NoneType' object has no attribute 'get'
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: Traceback (most recent call last):
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     res = self.dispatcher.dispatch(message)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 276, in dispatch
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return self._do_dispatch(endpoint, method, ctxt, args)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_messaging/rpc/dispatcher.py", line 196, in _do_dispatch
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     result = func(ctxt, **new_args)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 139, in wrapped
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     setattr(e, '_RETRY_EXCEEDED', True)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self.force_reraise()
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     six.reraise(self.type_, self.value, self.tb)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     raise value
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 135, in wrapped
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return f(*args, **kwargs)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 154, in wrapper
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     ectxt.value = e.inner_exc
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self.force_reraise()
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     six.reraise(self.type_, self.value, self.tb)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     raise value
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_db/api.py", line 142, in wrapper
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return f(*args, **kwargs)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 183, in wrapped
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     LOG.debug("Retry wrapper got retriable exception: %s", e)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self.force_reraise()
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     six.reraise(self.type_, self.value, self.tb)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/six.py", line 703, in reraise
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     raise value
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/usr/local/lib/python3.6/dist-packages/neutron_lib/db/api.py", line 179, in wrapped
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     return f(*dup_args, **dup_kwargs)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/api/rpc/handlers/l3_rpc.py", line 319, in get_agent_gateway_port
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     admin_ctx, network_id, host)
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_dvr_db.py", line 1078, in create_fip_agent_gw_port_if_not_exists
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     self._populate_mtu_and_subnets_for_ports(context, [agent_port])
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1762, in _populate_mtu_and_subnets_for_ports
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     for p in self._each_port_having_fixed_ips(ports)]
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1761, in <listcomp>
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     network_ids = [p['network_id']
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:   File "/opt/stack/neutron/neutron/db/l3_db.py", line 1709, in _each_port_having_fixed_ips
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]:     fixed_ips = port.get('fixed_ips', [])
  Mar 02 11:42:59 devstack-ubuntu-ovs neutron-l3-agent[23633]: AttributeError: 'NoneType' object has no attribute 'get'

  
  To solve this issue You need to manually remove corresponding row from "dvr_fip_gateway_port_network" db table.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1866336/+subscriptions