yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1615390] [NEW] neutron-l3-agent traceback while updating router

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Serguei Bezverkhi <sbezverk@xxxxxxxxx>
Date: Sun, 21 Aug 2016 15:11:01 -0000
Reply-to: Bug 1615390 <1615390@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

With the latest master I see this traceback over and over in the log.

2016-08-21 11:08:24.256 1 DEBUG neutron.agent.l3.agent [-] Starting router update for baa3335b-0013-42dd-856a-64a5c2557a01, action None, priority 1 _process_router_update /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py:461
2016-08-21 11:08:24.258 1 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 0745147d6f0d4b3380c0da2d53c9909a exchange 'neutron' topic 'q-l3-plugin' _send /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-08-21 11:08:24.564 1 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 0745147d6f0d4b3380c0da2d53c9909a __call__ /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-08-21 11:08:24.565 1 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-ex'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
2016-08-21 11:08:24.724 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:140
2016-08-21 11:08:24.725 1 DEBUG neutron.callbacks.manager [-] Notify callbacks for router, before_update _notify_loop /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/callbacks/manager.py:140
2016-08-21 11:08:24.726 1 DEBUG neutron.agent.l3.router_info [-] process router updates process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1036
2016-08-21 11:08:24.727 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-baa3335b-0013-42dd-856a-64a5c2557a01', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
2016-08-21 11:08:24.988 1 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument

2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 242, in call
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1037, in process
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     self._process_internal_ports(agent.pd)
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 543, in _process_internal_ports
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     existing_devices = self._get_existing_devices()
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 426, in _get_existing_devices
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 131, in get_devices
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     raise RuntimeError(msg)
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info 
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info 
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'baa3335b-0013-42dd-856a-64a5c2557a01'
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 495, in _process_router_update
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 432, in _process_router_if_compatible
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 447, in _process_updated_router
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     ri.process(self)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 245, in call
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self.logger(e)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self.force_reraise()
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 242, in call
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1037, in process
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self._process_internal_ports(agent.pd)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 543, in _process_internal_ports
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     existing_devices = self._get_existing_devices()
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 426, in _get_existing_devices
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 131, in get_devices
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     raise RuntimeError(msg)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent 
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent 
2016-08-21 11:08:24.992 1 DEBUG neutron.agent.l3.agent [-] Starting router update for baa3335b-0013-42dd-856a-64a5c2557a01, action None, priority 1 _process_router_update /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py:461
2016-08-21 11:08:24.993 1 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: ded128bbbae5422ea5ddc3a2fda4ae1d exchange 'neutron' topic 'q-l3-plugin' _send /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1615390

Title:
  neutron-l3-agent traceback while updating router

Status in neutron:
  New

Bug description:
  With the latest master I see this traceback over and over in the log.

  2016-08-21 11:08:24.256 1 DEBUG neutron.agent.l3.agent [-] Starting router update for baa3335b-0013-42dd-856a-64a5c2557a01, action None, priority 1 _process_router_update /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py:461
  2016-08-21 11:08:24.258 1 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 0745147d6f0d4b3380c0da2d53c9909a exchange 'neutron' topic 'q-l3-plugin' _send /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448
  2016-08-21 11:08:24.564 1 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 0745147d6f0d4b3380c0da2d53c9909a __call__ /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296
  2016-08-21 11:08:24.565 1 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-ex'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
  2016-08-21 11:08:24.724 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:140
  2016-08-21 11:08:24.725 1 DEBUG neutron.callbacks.manager [-] Notify callbacks for router, before_update _notify_loop /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/callbacks/manager.py:140
  2016-08-21 11:08:24.726 1 DEBUG neutron.agent.l3.router_info [-] process router updates process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1036
  2016-08-21 11:08:24.727 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-baa3335b-0013-42dd-856a-64a5c2557a01', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
  2016-08-21 11:08:24.988 1 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument

  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 242, in call
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1037, in process
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     self._process_internal_ports(agent.pd)
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 543, in _process_internal_ports
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     existing_devices = self._get_existing_devices()
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 426, in _get_existing_devices
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 131, in get_devices
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info     raise RuntimeError(msg)
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info 
  2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info 
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'baa3335b-0013-42dd-856a-64a5c2557a01'
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 495, in _process_router_update
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 432, in _process_router_if_compatible
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 447, in _process_updated_router
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     ri.process(self)
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 245, in call
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self.logger(e)
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self.force_reraise()
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 242, in call
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1037, in process
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     self._process_internal_ports(agent.pd)
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 543, in _process_internal_ports
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     existing_devices = self._get_existing_devices()
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 426, in _get_existing_devices
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 131, in get_devices
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent     raise RuntimeError(msg)
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent 
  2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent 
  2016-08-21 11:08:24.992 1 DEBUG neutron.agent.l3.agent [-] Starting router update for baa3335b-0013-42dd-856a-64a5c2557a01, action None, priority 1 _process_router_update /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py:461
  2016-08-21 11:08:24.993 1 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: ded128bbbae5422ea5ddc3a2fda4ae1d exchange 'neutron' topic 'q-l3-plugin' _send /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1615390/+subscriptions
Follow ups

[Bug 1615390] Re: neutron-l3-agent traceback while updating router
From: Launchpad Bug Tracker, 2016-11-12