yahoo-eng-team team mailing list archive

[OpenStack Infra <1761260@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/558898
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=ed7dd4dae0cf80f2314d9e113995e63a362b8295
Submitter: Zuul
Branch:    master

commit ed7dd4dae0cf80f2314d9e113995e63a362b8295
Author: Swaminathan Vasudevan <SVasudevan@xxxxxxxx>
Date:   Wed Apr 4 11:33:56 2018 -0700

    DVR: Check for item_allocator key before releasing
    
    item_allocator class maintains the allocation of
    items(link local address) from a given pool.
    There may be cases where the key has been already
    released or not being remembered while trying to
    release it.
    So it makes sense to lookup for the key before
    releasing the key.
    
    Change-Id: I34765dd3efa6c6742caeb7f6ae2ff14009dce6dd
    Closes-Bug: #1761260


** Changed in: neutron
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1761260

Title:
  DVR: Add a check for the item_allocator IP before trying to release
  it, since we see a KeyError sometimes, when the item is not there
  anymore.

Status in neutron:
  Fix Released

Bug description:
  We have seen this Traceback in Pike based installation, while trying
  to cleanup a gateway with DVR routers.

  2018-04-03 20:30:10.081 9672 DEBUG neutron.agent.l3.dvr_fip_ns [-] Delete FIP link interfaces for router: e415276a-4f37-4ee0-ba48-12d3909153c7 delete_rtr_2_fip_link /opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-pac
  kages/neutron/agent/l3/dvr_fip_ns.py:364
  2018-04-03 20:30:10.082 9672 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-e415276a-4f37-4ee0-ba48-12d3909153c7', 'ip', '-o', 'link', 'show', 'rfp-e415276a-4'] execute_ro
  otwrap_daemon /opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info [-] u'e415276a-4f37-4ee0-ba48-12d3909153c7': KeyError: u'e415276a-4f37-4ee0-ba48-12d3909153c7'
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/common/utils.py", line 186, in call
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1118, in process_delete
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self._process_external_on_delete()
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 890, in _process_external_on_delete
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self._process_external_gateway(ex_gw_port)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 799, in _process_external_gateway
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.external_gateway_removed(self.ex_gw_port, interface_name)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 513, in external_gateway_removed
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.fip_ns.delete_rtr_2_fip_link(self)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 402, in delete_rtr_2_fip_link
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.local_subnets.release(ri.router_id)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/item_allocator.py", line 116, in release
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.pool.add(self.allocations.pop(key))
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info KeyError: u'e415276a-4f37-4ee0-ba48-12d3909153c7'
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info

  Probably a check to make sure if the Key exists before release would
  be a good idea.

  We might also see if we can reproduce this in the master branch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1761260/+subscriptions