yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1573073] Re: When router has no ports _process_updated_router fails because the namespace does not exist

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Corey Bryant <corey.bryant@xxxxxxxxxxxxx>
Date: Fri, 04 Nov 2016 12:25:04 -0000
Reply-to: Bug 1573073 <1573073@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: neutron (Ubuntu)
   Importance: Undecided
       Status: New

** Also affects: neutron (Ubuntu Zesty)
   Importance: Undecided
       Status: New

** Also affects: neutron (Ubuntu Xenial)
   Importance: Undecided
       Status: New

** Also affects: neutron (Ubuntu Yakkety)
   Importance: Undecided
       Status: New

** Also affects: cloud-archive
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/liberty
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/newton
   Importance: Undecided
       Status: New

** Also affects: cloud-archive/mitaka
   Importance: Undecided
       Status: New

** Changed in: cloud-archive/liberty
       Status: New => Triaged

** Changed in: cloud-archive/liberty
   Importance: Undecided => High

** Changed in: cloud-archive/mitaka
       Status: New => Triaged

** Changed in: cloud-archive/mitaka
   Importance: Undecided => High

** Changed in: cloud-archive/newton
   Importance: Undecided => High

** Changed in: neutron (Ubuntu Zesty)
   Importance: Undecided => High

** Changed in: neutron (Ubuntu Yakkety)
   Importance: Undecided => High

** Changed in: neutron (Ubuntu Xenial)
   Importance: Undecided => High

** Changed in: cloud-archive/newton
       Status: New => Triaged

** Changed in: neutron (Ubuntu Xenial)
       Status: New => Triaged

** Changed in: neutron (Ubuntu Zesty)
       Status: New => Triaged

** Changed in: neutron (Ubuntu Yakkety)
       Status: New => Triaged

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1573073

Title:
  When router has no ports _process_updated_router fails because the
  namespace does not exist

Status in Ubuntu Cloud Archive:
  Triaged
Status in Ubuntu Cloud Archive liberty series:
  Triaged
Status in Ubuntu Cloud Archive mitaka series:
  Triaged
Status in Ubuntu Cloud Archive newton series:
  Triaged
Status in neutron:
  In Progress
Status in neutron package in Ubuntu:
  Triaged
Status in neutron source package in Xenial:
  Triaged
Status in neutron source package in Yakkety:
  Triaged
Status in neutron source package in Zesty:
  Triaged

Bug description:
  Happens in Kilo. Cannot test on other releases.

  Steps to reproduce:

  1) create a router and set at least a port, also the gateway is fine
  2) check that the namespace exists with
     ip netns show | grep qrouter-<uuid>
  3) check the ports are there
     ip netns exec qrouter-<uuid> ip addr show
  4) delete all ports from the router
  5) check that only loopback interface is present
     ip netns exec qrouter-<uuid> ip addr show
  6) run the cronjob task that is installed in the file
     /etc/cron.d/neutron-l3-agent-netns-cleanup
      so basically run this command:
     /usr/bin/neutron-netns-cleanup --config-file=/etc/neutron/neutron.conf --config-file=/etc/neutron/l3_agent.ini
  7) the namespace should be gone:
     ip netns show | grep qrouter-<uuid>
  8) delete the neutron router.
  9) check log file /var/log/neutron/vpn-agent.log

  
  When the router has no ports the namespace is deleted from the network node by the cronjob. However this brakes the router updates and the file vpn-agent.log is flooded with this traces:

  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Traceback (most recent call last):
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     return func(*args, **kwargs)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     self._process_internal_ports()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info     raise RuntimeError(m)
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info RuntimeError:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Exit code: 1
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdin:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stdout:
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.771 23382 TRACE neutron.agent.l3.router_info
  2016-04-21 16:22:17.774 23382 ERROR neutron.agent.l3.agent [-] Failed to process compatible router '8fc0f640-35bb-4d0b-bbbd-80c22be0e762'
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 467, in _process_router_update
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 414, in _process_router_if_compatible
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_updated_router(router)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/agent.py", line 428, in _process_updated_router
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ri.process(self)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 346, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self.logger(e)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 343, in call
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     return func(*args, **kwargs)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 628, in process
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     self._process_internal_ports()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 404, in _process_internal_ports
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     existing_devices = self._get_existing_devices()
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 328, in _get_existing_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 102, in get_devices
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 137, in execute
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent     raise RuntimeError(m)
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent RuntimeError:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f ']
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Exit code: 1
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdin:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stdout:
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent Stderr: Cannot open network namespace "qrouter-8fc0f640-35bb-4d0b-bbbd-80c22be0e762": No such file or directory
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent
  2016-04-21 16:22:17.774 23382 TRACE neutron.agent.l3.agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1573073/+subscriptions
References

[Bug 1573073] [NEW] When router has no ports _process_updated_router fails because the namespace does not exist
From: Saverio Proto, 2016-04-21