← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #72138

[Bug 1761260] [NEW] DVR: Add a check for the item_allocator IP before trying to release it, since we see a KeyError sometimes, when the item is not there anymore.

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

We have seen this Traceback in Pike based installation, while trying to
cleanup a gateway with DVR routers.

2018-04-03 20:30:10.081 9672 DEBUG neutron.agent.l3.dvr_fip_ns [-] Delete FIP link interfaces for router: e415276a-4f37-4ee0-ba48-12d3909153c7 delete_rtr_2_fip_link /opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-pac
kages/neutron/agent/l3/dvr_fip_ns.py:364
2018-04-03 20:30:10.082 9672 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-e415276a-4f37-4ee0-ba48-12d3909153c7', 'ip', '-o', 'link', 'show', 'rfp-e415276a-4'] execute_ro
otwrap_daemon /opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info [-] u'e415276a-4f37-4ee0-ba48-12d3909153c7': KeyError: u'e415276a-4f37-4ee0-ba48-12d3909153c7'
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/common/utils.py", line 186, in call
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1118, in process_delete
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self._process_external_on_delete()
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 890, in _process_external_on_delete
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self._process_external_gateway(ex_gw_port)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 799, in _process_external_gateway
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.external_gateway_removed(self.ex_gw_port, interface_name)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 513, in external_gateway_removed
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.fip_ns.delete_rtr_2_fip_link(self)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 402, in delete_rtr_2_fip_link
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.local_subnets.release(ri.router_id)
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/item_allocator.py", line 116, in release
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.pool.add(self.allocations.pop(key))
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info KeyError: u'e415276a-4f37-4ee0-ba48-12d3909153c7'
2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info

Probably a check to make sure if the Key exists before release would be
a good idea.

We might also see if we can reproduce this in the master branch.

** Affects: neutron
     Importance: Low
         Status: New


** Tags: l3-dvr-backlog

** Changed in: neutron
   Importance: Undecided => Critical

** Changed in: neutron
   Importance: Critical => Low

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1761260

Title:
  DVR: Add a check for the item_allocator IP before trying to release
  it, since we see a KeyError sometimes, when the item is not there
  anymore.

Status in neutron:
  New

Bug description:
  We have seen this Traceback in Pike based installation, while trying
  to cleanup a gateway with DVR routers.

  2018-04-03 20:30:10.081 9672 DEBUG neutron.agent.l3.dvr_fip_ns [-] Delete FIP link interfaces for router: e415276a-4f37-4ee0-ba48-12d3909153c7 delete_rtr_2_fip_link /opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-pac
  kages/neutron/agent/l3/dvr_fip_ns.py:364
  2018-04-03 20:30:10.082 9672 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-e415276a-4f37-4ee0-ba48-12d3909153c7', 'ip', '-o', 'link', 'show', 'rfp-e415276a-4'] execute_ro
  otwrap_daemon /opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/linux/utils.py:108
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info [-] u'e415276a-4f37-4ee0-ba48-12d3909153c7': KeyError: u'e415276a-4f37-4ee0-ba48-12d3909153c7'
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/common/utils.py", line 186, in call
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1118, in process_delete
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self._process_external_on_delete()
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 890, in _process_external_on_delete
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self._process_external_gateway(ex_gw_port)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 799, in _process_external_gateway
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.external_gateway_removed(self.ex_gw_port, interface_name)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_local_router.py", line 513, in external_gateway_removed
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.fip_ns.delete_rtr_2_fip_link(self)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/dvr_fip_ns.py", line 402, in delete_rtr_2_fip_link
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.local_subnets.release(ri.router_id)
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info   File "/opt/stack/venv/neutron-20180328T152147Z/lib/python2.7/site-packages/neutron/agent/l3/item_allocator.py", line 116, in release
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info     self.pool.add(self.allocations.pop(key))
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info KeyError: u'e415276a-4f37-4ee0-ba48-12d3909153c7'
  2018-04-03 20:30:10.179 9672 ERROR neutron.agent.l3.router_info

  Probably a check to make sure if the Key exists before release would
  be a good idea.

  We might also see if we can reproduce this in the master branch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1761260/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next