yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1606844] [NEW] Neutron constantly resyncing deleted router

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Oleg Bondarev <obondarev@xxxxxxxxxxxx>
Date: Wed, 27 Jul 2016 09:41:09 -0000
Reply-to: Bug 1606844 <1606844@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

No need to constantly resync router which was deleted and for which
there is no namespace.

Observed: l3 agent log full of

2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent [-] Error while deleting router 81ef46de-f7f9-4c5e-b787-c935e0af253a
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 359, in _safe_router_removed
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self._router_removed(router_id)
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 377, in _router_removed
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     ri.delete(self)
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 347, in delete
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self.process_delete(agent)
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 385, in call
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self.logger(e)
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self.force_reraise()
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 382, in call
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 947, in process_delete
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self._process_internal_ports(agent.pd)
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 530, in _process_internal_ports
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     existing_devices = self._get_existing_devices()
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 413, in _get_existing_devices
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 130, in get_devices
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     raise RuntimeError(msg)
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-81ef46de-f7f9-4c5e-b787-c935e0af253a": No such file or directory
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent
2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent
2016-07-26 14:00:45.236 13360 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-81ef46de-f7f9-4c5e-b787-c935e0af253a": No such file or directory

this consumes memory, cpu, disk.

** Affects: neutron
     Importance: Undecided
     Assignee: Oleg Bondarev (obondarev)
         Status: New


** Tags: l3-ipam-dhcp

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1606844

Title:
  Neutron constantly resyncing deleted router

Status in neutron:
  New

Bug description:
  No need to constantly resync router which was deleted and for which
  there is no namespace.

  Observed: l3 agent log full of

  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent [-] Error while deleting router 81ef46de-f7f9-4c5e-b787-c935e0af253a
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 359, in _safe_router_removed
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self._router_removed(router_id)
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 377, in _router_removed
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     ri.delete(self)
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 347, in delete
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self.process_delete(agent)
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 385, in call
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self.logger(e)
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self.force_reraise()
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 382, in call
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 947, in process_delete
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     self._process_internal_ports(agent.pd)
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 530, in _process_internal_ports
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     existing_devices = self._get_existing_devices()
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 413, in _get_existing_devices
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 130, in get_devices
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent     raise RuntimeError(msg)
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-81ef46de-f7f9-4c5e-b787-c935e0af253a": No such file or directory
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent
  2016-07-26 14:00:45.224 13360 ERROR neutron.agent.l3.agent
  2016-07-26 14:00:45.236 13360 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-81ef46de-f7f9-4c5e-b787-c935e0af253a": No such file or directory

  this consumes memory, cpu, disk.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1606844/+subscriptions
Follow ups

[Bug 1606844] Re: L3 agent constantly resyncing deleted router
From: OpenStack Infra, 2016-08-09