yahoo-eng-team team mailing list archive

[Alan Pevec <1448148@xxxxxxxxxxxxxxxxxx>]

** Also affects: neutron/kilo
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1448148

Title:
  Device not found: exceptions in l3 grenade

Status in neutron:
  Fix Released
Status in neutron kilo series:
  New

Bug description:
  exceptions have been seen in screen-q-vpn.txt.gz at the gate, and it
  happens quite often recently:

       2015-04-23 15:55:56.236 ERROR neutron.agent.l3.agent [req-5ea4d9d1-66ab-444c-a66f-c48094f3582d None None] Failed to process compatible router 'aeb00076-1c9e-431d-973f-ce1123c918a7'
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Traceback (most recent call last):
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/l3/agent.py", line 452, in _process_router_update
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/l3/agent.py", line 404, in _process_router_if_compatible
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     self._process_added_router(router)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/l3/agent.py", line 412, in _process_added_router
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     ri.process(self)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/common/utils.py", line 346, in call
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     self.logger(e)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/common/utils.py", line 343, in call
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     return func(*args, **kwargs)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 605, in process
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     self._process_internal_ports()
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 361, in _process_internal_ports
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     self.internal_network_added(p)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 312, in internal_network_added
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     INTERNAL_DEV_PREFIX)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 288, in _internal_network_added
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     prefix=prefix)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/linux/interface.py", line 264, in plug
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     ns_dev.link.set_up()
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 276, in set_up
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     self._as_root([], ('set', self.name, 'up'))
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 222, in _as_root
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     use_root_namespace=use_root_namespace)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 69, in _as_root
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 78, in _execute
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     log_fail_as_error=log_fail_as_error)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent   File "/opt/stack/old/neutron/neutron/agent/linux/utils.py", line 137, in execute
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent     raise RuntimeError(m)
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent RuntimeError: 
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Command: ['ip', 'netns', 'exec', u'qrouter-aeb00076-1c9e-431d-973f-ce1123c918a7', 'ip', 'link', 'set', u'qr-91aaad43-1e', 'up']
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Exit code: 1
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Stdin: 
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Stdout: 
  2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Stderr: Cannot find device "qr-91aaad43-1e"

  Do a search at
  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIkNhbm5vdCBmaW5kIGRldmljZVwiIEFORCBtZXNzYWdlOiBcInFyLVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI5ODgzMDUxOTIxfQ==

  With the search string: message: "Cannot find device" AND message:
  "qr-", many occurrences have been seen.

  This seems to be the reason for failures of check-grenade-dvm-neutron
  as seen in this review: https://review.openstack.org/#/c/176041/
  although not of the test cases failed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1448148/+subscriptions