yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #32350
[Bug 1448148] [NEW] Device not found: exceptions in l3 grenade
Public bug reported:
exceptions have been seen in screen-q-vpn.txt.gz at the gate, and it
happens quite often recently:
2015-04-23 15:55:56.236 ERROR neutron.agent.l3.agent [req-5ea4d9d1-66ab-444c-a66f-c48094f3582d None None] Failed to process compatible router 'aeb00076-1c9e-431d-973f-ce1123c918a7'
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/agent.py", line 452, in _process_router_update
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self._process_router_if_compatible(router)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/agent.py", line 404, in _process_router_if_compatible
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self._process_added_router(router)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/agent.py", line 412, in _process_added_router
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent ri.process(self)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/common/utils.py", line 346, in call
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self.logger(e)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/common/utils.py", line 343, in call
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent return func(*args, **kwargs)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 605, in process
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self._process_internal_ports()
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 361, in _process_internal_ports
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self.internal_network_added(p)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 312, in internal_network_added
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent INTERNAL_DEV_PREFIX)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 288, in _internal_network_added
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent prefix=prefix)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/interface.py", line 264, in plug
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent ns_dev.link.set_up()
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 276, in set_up
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self._as_root([], ('set', self.name, 'up'))
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 222, in _as_root
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent use_root_namespace=use_root_namespace)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 69, in _as_root
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 78, in _execute
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent log_fail_as_error=log_fail_as_error)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/utils.py", line 137, in execute
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent raise RuntimeError(m)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent RuntimeError:
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Command: ['ip', 'netns', 'exec', u'qrouter-aeb00076-1c9e-431d-973f-ce1123c918a7', 'ip', 'link', 'set', u'qr-91aaad43-1e', 'up']
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Exit code: 1
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Stdin:
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Stdout:
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Stderr: Cannot find device "qr-91aaad43-1e"
Do a search at
http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIkNhbm5vdCBmaW5kIGRldmljZVwiIEFORCBtZXNzYWdlOiBcInFyLVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI5ODgzMDUxOTIxfQ==
With the search string: message: "Cannot find device" AND message:
"qr-", many occurrences have been seen.
This seems to be the reason for failures of check-grenade-dvm-neutron as
seen in this review: https://review.openstack.org/#/c/176041/ although
not of the test cases failed.
** Affects: neutron
Importance: Undecided
Assignee: Baodong (Robert) Li (baoli)
Status: New
** Changed in: neutron
Assignee: (unassigned) => Baodong (Robert) Li (baoli)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1448148
Title:
Device not found: exceptions in l3 grenade
Status in OpenStack Neutron (virtual network service):
New
Bug description:
exceptions have been seen in screen-q-vpn.txt.gz at the gate, and it
happens quite often recently:
2015-04-23 15:55:56.236 ERROR neutron.agent.l3.agent [req-5ea4d9d1-66ab-444c-a66f-c48094f3582d None None] Failed to process compatible router 'aeb00076-1c9e-431d-973f-ce1123c918a7'
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Traceback (most recent call last):
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/agent.py", line 452, in _process_router_update
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self._process_router_if_compatible(router)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/agent.py", line 404, in _process_router_if_compatible
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self._process_added_router(router)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/agent.py", line 412, in _process_added_router
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent ri.process(self)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/common/utils.py", line 346, in call
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self.logger(e)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/common/utils.py", line 343, in call
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent return func(*args, **kwargs)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 605, in process
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self._process_internal_ports()
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 361, in _process_internal_ports
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self.internal_network_added(p)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 312, in internal_network_added
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent INTERNAL_DEV_PREFIX)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/l3/router_info.py", line 288, in _internal_network_added
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent prefix=prefix)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/interface.py", line 264, in plug
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent ns_dev.link.set_up()
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 276, in set_up
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent self._as_root([], ('set', self.name, 'up'))
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 222, in _as_root
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent use_root_namespace=use_root_namespace)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 69, in _as_root
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/ip_lib.py", line 78, in _execute
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent log_fail_as_error=log_fail_as_error)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent File "/opt/stack/old/neutron/neutron/agent/linux/utils.py", line 137, in execute
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent raise RuntimeError(m)
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent RuntimeError:
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Command: ['ip', 'netns', 'exec', u'qrouter-aeb00076-1c9e-431d-973f-ce1123c918a7', 'ip', 'link', 'set', u'qr-91aaad43-1e', 'up']
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Exit code: 1
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Stdin:
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Stdout:
2015-04-23 15:55:56.236 3736 TRACE neutron.agent.l3.agent Stderr: Cannot find device "qr-91aaad43-1e"
Do a search at
http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOiBcIkNhbm5vdCBmaW5kIGRldmljZVwiIEFORCBtZXNzYWdlOiBcInFyLVwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDI5ODgzMDUxOTIxfQ==
With the search string: message: "Cannot find device" AND message:
"qr-", many occurrences have been seen.
This seems to be the reason for failures of check-grenade-dvm-neutron
as seen in this review: https://review.openstack.org/#/c/176041/
although not of the test cases failed.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1448148/+subscriptions
Follow ups
References