← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #54303

[Bug 1606801] [NEW] deleting router run into race condition

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

After deleting a router the logfiles of both network nodes are filled up with " RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-3767"
After i have restarted the openstack services on the network nodes, no new entries

Reproduceable: yes

Steps:
* add router via CLI or dashboard
* delete router via CLI or dashboard
* logfiles grow up 

Openstack version: mitaka ( this error occured on liberty too ! )

OS: Centos 7, latest updates

Installed Packages on nerwork nodes
openstack-neutron-vpnaas-8.0.0-1.el7.noarch
openstack-neutron-common-8.1.2-1.el7.noarch
openstack-neutron-metering-agent-8.1.2-1.el7.noarch
python-neutronclient-4.1.1-2.el7.noarch
python-neutron-8.1.2-1.el7.noarch
python-neutron-fwaas-8.0.0-3.el7.noarch
openstack-neutron-ml2-8.1.2-1.el7.noarch
openstack-neutron-bgp-dragent-8.1.2-1.el7.noarch
python-neutron-vpnaas-8.0.0-1.el7.noarch
openstack-neutron-openvswitch-8.1.2-1.el7.noarch
openstack-neutron-8.1.2-1.el7.noarch
python-neutron-lib-0.0.2-1.el7.noarch
openstack-neutron-fwaas-8.0.0-3.el7.noarch


Logfile network node:
2.770 44778 DEBUG neutron.agent.linux.ra [-] radvd disabled for router 37678766-597a-4e33-b83a-65142ca2ced8 disable /usr/lib/python2.7/site-packages/neutron/agent/linux/ra.py:190
2016-07-27 09:10:02.770 44778 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-37678766-597a-4e33-b83a-65142ca2ced8', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-37678766-597a-4e33-b83a-65142ca2ced8": No such file or directory
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent [-] Error while deleting router 37678766-597a-4e33-b83a-65142ca2ced8
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 359, in _safe_router_removed
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     self._router_removed(router_id)
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 377, in _router_removed
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     ri.delete(self)
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 380, in delete
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     super(HaRouter, self).delete(agent)
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 349, in delete
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     self.router_namespace.delete()
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/namespaces.py", line 100, in delete
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     for d in ns_ip.get_devices(exclude_loopback=True):
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 130, in get_devices
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     raise RuntimeError(msg)
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-37678766-597a-4e33-b83a-65142ca2ced8": No such file or directory
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent


Attached logfiles of control node and both network nodes.
At 09:09:00 ->  added router
At 09:10:00 -> deleted router

** Affects: neutron
     Importance: Undecided
         Status: New

** Attachment added: "logs.tar.gz"
   https://bugs.launchpad.net/bugs/1606801/+attachment/4707969/+files/logs.tar.gz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1606801

Title:
  deleting router run into race condition

Status in neutron:
  New

Bug description:
  After deleting a router the logfiles of both network nodes are filled up with " RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-3767"
  After i have restarted the openstack services on the network nodes, no new entries

  Reproduceable: yes

  Steps:
  * add router via CLI or dashboard
  * delete router via CLI or dashboard
  * logfiles grow up 

  Openstack version: mitaka ( this error occured on liberty too ! )

  OS: Centos 7, latest updates

  Installed Packages on nerwork nodes
  openstack-neutron-vpnaas-8.0.0-1.el7.noarch
  openstack-neutron-common-8.1.2-1.el7.noarch
  openstack-neutron-metering-agent-8.1.2-1.el7.noarch
  python-neutronclient-4.1.1-2.el7.noarch
  python-neutron-8.1.2-1.el7.noarch
  python-neutron-fwaas-8.0.0-3.el7.noarch
  openstack-neutron-ml2-8.1.2-1.el7.noarch
  openstack-neutron-bgp-dragent-8.1.2-1.el7.noarch
  python-neutron-vpnaas-8.0.0-1.el7.noarch
  openstack-neutron-openvswitch-8.1.2-1.el7.noarch
  openstack-neutron-8.1.2-1.el7.noarch
  python-neutron-lib-0.0.2-1.el7.noarch
  openstack-neutron-fwaas-8.0.0-3.el7.noarch

  
  Logfile network node:
  2.770 44778 DEBUG neutron.agent.linux.ra [-] radvd disabled for router 37678766-597a-4e33-b83a-65142ca2ced8 disable /usr/lib/python2.7/site-packages/neutron/agent/linux/ra.py:190
  2016-07-27 09:10:02.770 44778 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-37678766-597a-4e33-b83a-65142ca2ced8', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-37678766-597a-4e33-b83a-65142ca2ced8": No such file or directory
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent [-] Error while deleting router 37678766-597a-4e33-b83a-65142ca2ced8
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 359, in _safe_router_removed
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     self._router_removed(router_id)
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 377, in _router_removed
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     ri.delete(self)
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 380, in delete
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     super(HaRouter, self).delete(agent)
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 349, in delete
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     self.router_namespace.delete()
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/namespaces.py", line 100, in delete
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     for d in ns_ip.get_devices(exclude_loopback=True):
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 130, in get_devices
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent     raise RuntimeError(msg)
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-37678766-597a-4e33-b83a-65142ca2ced8": No such file or directory
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent
  2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent


  Attached logfiles of control node and both network nodes.
  At 09:09:00 ->  added router
  At 09:10:00 -> deleted router

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1606801/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next