yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1654287] [NEW] functional test netns_cleanup failing in gate

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Daniel Alvarez <1654287@xxxxxxxxxxxxxxxxxx>
Date: Thu, 05 Jan 2017 14:25:48 -0000
Reply-to: Bug 1654287 <1654287@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:


The functional test for netns_cleanup has failed in the gate today [0].

Apparently, when trying to get the list of devices (ip_lib.get_devices()
'find /sys/class/net -maxdepth 1 -type 1 -printf %f') through
rootwrap_daemon, it's getting the output of the previous command instead
('netstat -nlp'). This causes that the netns_cleanup module tries to
unplug random devices which correspond to the actual output of the
'netstat' command.

This bug doesn't look related to the test itself but to rootwrap_daemon?
Maybe due to long output to the netstat command?


Relevant part of the log

2017-01-05 12:17:04.609 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'netstat', '-nlp'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
2017-01-05 12:17:04.613 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 0 execute neutron/agent/linux/utils.py:149
2017-01-05 12:17:04.614 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
2017-01-05 12:17:04.645 27615 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 14 __log_wakeup /opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
2017-01-05 12:17:04.686 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 0 execute neutron/agent/linux/utils.py:149
2017-01-05 12:17:04.688 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'ip', 'link', 'delete', 'Active'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
2017-01-05 12:17:04.746 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 0 execute neutron/agent/linux/utils.py:149
2017-01-05 12:17:04.747 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'ip', 'link', 'delete', 'Internet'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
2017-01-05 12:17:04.758 27615 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 14 __log_wakeup /opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
2017-01-05 12:17:04.815 27615 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 14 __log_wakeup /opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
2017-01-05 12:17:04.822 27615 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 7 __log_wakeup /opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
2017-01-05 12:17:04.822 27615 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): InterfaceToBridgeCommand(name=Internet) do_commit neutron/agent/ovsdb/impl_idl.py:100
2017-01-05 12:17:04.823 27615 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction aborted do_commit neutron/agent/ovsdb/impl_idl.py:124
2017-01-05 12:17:04.824 27615 DEBUG neutron.cmd.netns_cleanup [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Unable to find bridge for device: Internet unplug_device neutron/cmd/netns_cleanup.py:138
2017-01-05 12:17:04.824 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'ip', 'link', 'delete', 'connections'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
....
2017-01-05 12:17:06.388 27615 DEBUG neutron.cmd.netns_cleanup [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Unable to find bridge for device: Path unplug_device neutron/cmd/netns_cleanup.py:138
2017-01-05 12:17:06.389 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', '-o', 'netns', 'list'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
2017-01-05 12:17:06.454 27615 ERROR neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "Path"

2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Error unable to destroy namespace: qrouter-cf2030c6-c924-45bb-b13b-6774d275b394
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup Traceback (most recent call last):
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/cmd/netns_cleanup.py", line 250, in destroy_namespace
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     ip.garbage_collect_namespace()
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/agent/linux/ip_lib.py", line 222, in garbage_collect_namespace
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     if self.namespace and self.netns.exists(self.namespace):
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/agent/linux/ip_lib.py", line 888, in exists
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     run_as_root=cfg.CONF.AGENT.use_helper_for_ns_read)
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/agent/linux/ip_lib.py", line 107, in _execute
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     log_fail_as_error=log_fail_as_error)
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/agent/linux/utils.py", line 147, in execute
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     raise ProcessExecutionError(msg, returncode=returncode)
2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "Path"


[0] http://logs.openstack.org/51/396651/15/check/gate-neutron-dsvm-
functional-ubuntu-xenial/5d268f0/logs/dsvm-functional-
logs/neutron.tests.functional.cmd.test_netns_cleanup.NetnsCleanupTest.test_cleanup_network_namespaces_cleans_dhcp_and_l3_namespaces.txt.gz

** Affects: neutron
     Importance: Undecided
         Status: Confirmed


** Tags: functional-tests gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1654287

Title:
  functional test netns_cleanup failing in gate

Status in neutron:
  Confirmed

Bug description:
  
  The functional test for netns_cleanup has failed in the gate today [0].

  Apparently, when trying to get the list of devices
  (ip_lib.get_devices() 'find /sys/class/net -maxdepth 1 -type 1 -printf
  %f') through rootwrap_daemon, it's getting the output of the previous
  command instead ('netstat -nlp'). This causes that the netns_cleanup
  module tries to unplug random devices which correspond to the actual
  output of the 'netstat' command.

  This bug doesn't look related to the test itself but to
  rootwrap_daemon? Maybe due to long output to the netstat command?

  
  Relevant part of the log

  2017-01-05 12:17:04.609 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'netstat', '-nlp'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
  2017-01-05 12:17:04.613 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 0 execute neutron/agent/linux/utils.py:149
  2017-01-05 12:17:04.614 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
  2017-01-05 12:17:04.645 27615 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 14 __log_wakeup /opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
  2017-01-05 12:17:04.686 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 0 execute neutron/agent/linux/utils.py:149
  2017-01-05 12:17:04.688 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'ip', 'link', 'delete', 'Active'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
  2017-01-05 12:17:04.746 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 0 execute neutron/agent/linux/utils.py:149
  2017-01-05 12:17:04.747 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'ip', 'link', 'delete', 'Internet'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
  2017-01-05 12:17:04.758 27615 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 14 __log_wakeup /opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
  2017-01-05 12:17:04.815 27615 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 14 __log_wakeup /opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
  2017-01-05 12:17:04.822 27615 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 7 __log_wakeup /opt/stack/new/neutron/.tox/dsvm-functional/local/lib/python2.7/site-packages/ovs/poller.py:202
  2017-01-05 12:17:04.822 27615 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): InterfaceToBridgeCommand(name=Internet) do_commit neutron/agent/ovsdb/impl_idl.py:100
  2017-01-05 12:17:04.823 27615 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction aborted do_commit neutron/agent/ovsdb/impl_idl.py:124
  2017-01-05 12:17:04.824 27615 DEBUG neutron.cmd.netns_cleanup [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Unable to find bridge for device: Internet unplug_device neutron/cmd/netns_cleanup.py:138
  2017-01-05 12:17:04.824 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-cf2030c6-c924-45bb-b13b-6774d275b394', 'ip', 'link', 'delete', 'connections'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
  ....
  2017-01-05 12:17:06.388 27615 DEBUG neutron.cmd.netns_cleanup [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Unable to find bridge for device: Path unplug_device neutron/cmd/netns_cleanup.py:138
  2017-01-05 12:17:06.389 27615 DEBUG neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Running command (rootwrap daemon): ['ip', '-o', 'netns', 'list'] execute_rootwrap_daemon neutron/agent/linux/utils.py:108
  2017-01-05 12:17:06.454 27615 ERROR neutron.agent.linux.utils [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "Path"

  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup [req-68eceb29-052a-4c8c-8152-38bbe636cba5 - - - - -] Error unable to destroy namespace: qrouter-cf2030c6-c924-45bb-b13b-6774d275b394
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup Traceback (most recent call last):
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/cmd/netns_cleanup.py", line 250, in destroy_namespace
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     ip.garbage_collect_namespace()
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/agent/linux/ip_lib.py", line 222, in garbage_collect_namespace
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     if self.namespace and self.netns.exists(self.namespace):
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/agent/linux/ip_lib.py", line 888, in exists
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     run_as_root=cfg.CONF.AGENT.use_helper_for_ns_read)
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/agent/linux/ip_lib.py", line 107, in _execute
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     log_fail_as_error=log_fail_as_error)
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup   File "neutron/agent/linux/utils.py", line 147, in execute
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup     raise ProcessExecutionError(msg, returncode=returncode)
  2017-01-05 12:17:06.454 27615 ERROR neutron.cmd.netns_cleanup ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot find device "Path"


  [0] http://logs.openstack.org/51/396651/15/check/gate-neutron-dsvm-
  functional-ubuntu-xenial/5d268f0/logs/dsvm-functional-
  logs/neutron.tests.functional.cmd.test_netns_cleanup.NetnsCleanupTest.test_cleanup_network_namespaces_cleans_dhcp_and_l3_namespaces.txt.gz

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1654287/+subscriptions
Follow ups

[Bug 1654287] Re: rootwrap daemon may return output of previous command
From: OpenStack Infra, 2017-11-15
[Bug 1654287] Re: functional test netns_cleanup failing in gate
From: OpenStack Infra, 2017-01-20
[Bug 1654287] Re: functional test netns_cleanup failing in gate
From: Daniel Alvarez, 2017-01-18