← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #32109

[Bug 1446261] [NEW] gate-neutron-dsvm-functional race fails HA/DVR tests with network namespace not found

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

http://logs.openstack.org/21/174821/1/gate/gate-neutron-dsvm-
functional/eb6b441/console.html#_2015-04-20_07_41_14_649

This is happening quite often and is not just this specific test:

2015-04-20 07:41:14.649 | 2015-04-20 07:41:14.642 | {4} neutron.tests.functional.agent.test_l3_agent.L3HATestFramework.test_ha_router_failover [12.678377s] ... FAILED
2015-04-20 07:41:14.650 | 2015-04-20 07:41:14.643 | 
2015-04-20 07:41:14.670 | 2015-04-20 07:41:14.647 | Captured traceback:
2015-04-20 07:41:14.671 | 2015-04-20 07:41:14.649 | ~~~~~~~~~~~~~~~~~~~
2015-04-20 07:41:14.671 | 2015-04-20 07:41:14.650 |     Traceback (most recent call last):
2015-04-20 07:41:14.672 | 2015-04-20 07:41:14.652 |       File "neutron/tests/functional/agent/test_l3_agent.py", line 762, in test_ha_router_failover
2015-04-20 07:41:14.673 | 2015-04-20 07:41:14.653 |         ha_device.link.set_down()
2015-04-20 07:41:14.673 | 2015-04-20 07:41:14.655 |       File "neutron/agent/linux/ip_lib.py", line 279, in set_down
2015-04-20 07:41:14.674 | 2015-04-20 07:41:14.658 |         self._as_root([], ('set', self.name, 'down'))
2015-04-20 07:41:14.675 | 2015-04-20 07:41:14.661 |       File "neutron/agent/linux/ip_lib.py", line 222, in _as_root
2015-04-20 07:41:14.675 | 2015-04-20 07:41:14.663 |         use_root_namespace=use_root_namespace)
2015-04-20 07:41:14.676 | 2015-04-20 07:41:14.664 |       File "neutron/agent/linux/ip_lib.py", line 69, in _as_root
2015-04-20 07:41:14.677 | 2015-04-20 07:41:14.666 |         log_fail_as_error=self.log_fail_as_error)
2015-04-20 07:41:14.677 | 2015-04-20 07:41:14.668 |       File "neutron/agent/linux/ip_lib.py", line 78, in _execute
2015-04-20 07:41:14.679 | 2015-04-20 07:41:14.672 |         log_fail_as_error=log_fail_as_error)
2015-04-20 07:41:14.681 | 2015-04-20 07:41:14.675 |       File "neutron/agent/linux/utils.py", line 137, in execute
2015-04-20 07:41:14.683 | 2015-04-20 07:41:14.676 |         raise RuntimeError(m)
2015-04-20 07:41:14.684 | 2015-04-20 07:41:14.678 |     RuntimeError: 
2015-04-20 07:41:14.686 | 2015-04-20 07:41:14.679 |     Command: ['ip', 'netns', 'exec', 'qrouter-425dfc14-0f4d-45fa-8218-531cae21711f@agent1', 'ip', 'link', 'set', 'ha-29cbe060-37', 'down']
2015-04-20 07:41:14.688 | 2015-04-20 07:41:14.681 |     Exit code: 1
2015-04-20 07:41:14.689 | 2015-04-20 07:41:14.682 |     Stdin: 
2015-04-20 07:41:14.691 | 2015-04-20 07:41:14.684 |     Stdout: 
2015-04-20 07:41:14.692 | 2015-04-20 07:41:14.685 |     Stderr: Cannot open network namespace "qrouter-425dfc14-0f4d-45fa-8218-531cae21711f@agent1": No such file or directory


If you restrict to just the gate job, it's 62 hits in 7 days:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU3RkZXJyOiBDYW5ub3Qgb3BlbiBuZXR3b3JrIG5hbWVzcGFjZSBcXFwicXJvdXRlclwiIEFORCBtZXNzYWdlOlwiQGFnZW50MVxcXCI6IE5vIHN1Y2ggZmlsZSBvciBkaXJlY3RvcnlcIiBBTkQgYnVpbGRfbmFtZTpcImdhdGUtbmV1dHJvbi1kc3ZtLWZ1bmN0aW9uYWxcIiBBTkQgdGFnczpcImNvbnNvbGVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyOTU0MDU5NTI0NiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

** Affects: neutron
     Importance: High
         Status: Confirmed


** Tags: dvr ha testing

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1446261

Title:
  gate-neutron-dsvm-functional race fails HA/DVR tests with network
  namespace not found

Status in OpenStack Neutron (virtual network service):
  Confirmed

Bug description:
  http://logs.openstack.org/21/174821/1/gate/gate-neutron-dsvm-
  functional/eb6b441/console.html#_2015-04-20_07_41_14_649

  This is happening quite often and is not just this specific test:

  2015-04-20 07:41:14.649 | 2015-04-20 07:41:14.642 | {4} neutron.tests.functional.agent.test_l3_agent.L3HATestFramework.test_ha_router_failover [12.678377s] ... FAILED
  2015-04-20 07:41:14.650 | 2015-04-20 07:41:14.643 | 
  2015-04-20 07:41:14.670 | 2015-04-20 07:41:14.647 | Captured traceback:
  2015-04-20 07:41:14.671 | 2015-04-20 07:41:14.649 | ~~~~~~~~~~~~~~~~~~~
  2015-04-20 07:41:14.671 | 2015-04-20 07:41:14.650 |     Traceback (most recent call last):
  2015-04-20 07:41:14.672 | 2015-04-20 07:41:14.652 |       File "neutron/tests/functional/agent/test_l3_agent.py", line 762, in test_ha_router_failover
  2015-04-20 07:41:14.673 | 2015-04-20 07:41:14.653 |         ha_device.link.set_down()
  2015-04-20 07:41:14.673 | 2015-04-20 07:41:14.655 |       File "neutron/agent/linux/ip_lib.py", line 279, in set_down
  2015-04-20 07:41:14.674 | 2015-04-20 07:41:14.658 |         self._as_root([], ('set', self.name, 'down'))
  2015-04-20 07:41:14.675 | 2015-04-20 07:41:14.661 |       File "neutron/agent/linux/ip_lib.py", line 222, in _as_root
  2015-04-20 07:41:14.675 | 2015-04-20 07:41:14.663 |         use_root_namespace=use_root_namespace)
  2015-04-20 07:41:14.676 | 2015-04-20 07:41:14.664 |       File "neutron/agent/linux/ip_lib.py", line 69, in _as_root
  2015-04-20 07:41:14.677 | 2015-04-20 07:41:14.666 |         log_fail_as_error=self.log_fail_as_error)
  2015-04-20 07:41:14.677 | 2015-04-20 07:41:14.668 |       File "neutron/agent/linux/ip_lib.py", line 78, in _execute
  2015-04-20 07:41:14.679 | 2015-04-20 07:41:14.672 |         log_fail_as_error=log_fail_as_error)
  2015-04-20 07:41:14.681 | 2015-04-20 07:41:14.675 |       File "neutron/agent/linux/utils.py", line 137, in execute
  2015-04-20 07:41:14.683 | 2015-04-20 07:41:14.676 |         raise RuntimeError(m)
  2015-04-20 07:41:14.684 | 2015-04-20 07:41:14.678 |     RuntimeError: 
  2015-04-20 07:41:14.686 | 2015-04-20 07:41:14.679 |     Command: ['ip', 'netns', 'exec', 'qrouter-425dfc14-0f4d-45fa-8218-531cae21711f@agent1', 'ip', 'link', 'set', 'ha-29cbe060-37', 'down']
  2015-04-20 07:41:14.688 | 2015-04-20 07:41:14.681 |     Exit code: 1
  2015-04-20 07:41:14.689 | 2015-04-20 07:41:14.682 |     Stdin: 
  2015-04-20 07:41:14.691 | 2015-04-20 07:41:14.684 |     Stdout: 
  2015-04-20 07:41:14.692 | 2015-04-20 07:41:14.685 |     Stderr: Cannot open network namespace "qrouter-425dfc14-0f4d-45fa-8218-531cae21711f@agent1": No such file or directory

  
  If you restrict to just the gate job, it's 62 hits in 7 days:

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiU3RkZXJyOiBDYW5ub3Qgb3BlbiBuZXR3b3JrIG5hbWVzcGFjZSBcXFwicXJvdXRlclwiIEFORCBtZXNzYWdlOlwiQGFnZW50MVxcXCI6IE5vIHN1Y2ggZmlsZSBvciBkaXJlY3RvcnlcIiBBTkQgYnVpbGRfbmFtZTpcImdhdGUtbmV1dHJvbi1kc3ZtLWZ1bmN0aW9uYWxcIiBBTkQgdGFnczpcImNvbnNvbGVcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQyOTU0MDU5NTI0NiwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1446261/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next