yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1528977] Re: Neutron router not working with latest iproute2 package included in CentOS-7.2-1511

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrew Poltavchenko <andreipoltavchenko@xxxxxxxxx>
Date: Thu, 24 Dec 2015 01:16:26 -0000
Reply-to: Bug 1528977 <1528977@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: centos
   Importance: Undecided
       Status: New

** No longer affects: centos

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1528977

Title:
  Neutron router not working with latest iproute2 package included in
  CentOS-7.2-1511

Status in neutron:
  New

Bug description:
  Seems that something has been changed in the new iproute version and
  now attempts to add more than one interface to router cause errors
  posted at the bottom. This affects neutron-l3-agent on CentOS-7.2-1511
  and possible on RedHat (cannot check this).

  Quick solution is to simple downgrade the package:
  # wget http://mirror.centos.org/centos/7.1.1503/os/x86_64/Packages/iproute-3.10.0-21.el7.x86_64.rpm
  # yum -y downgrade ./iproute-3.10.0-21.el7.x86_64.rpm

  Part of the /var/log/neutron/l3-agent.log:

  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.linux.utils [-]
  Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a']
  Exit code: 1
  Stdin:
  Stdout:
  Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists

  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info [-]
  Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a']
  Exit code: 1
  Stdin:
  Stdout:
  Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 356, in call
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     return func(*args, **kwargs)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 692, in process
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     self._process_internal_ports(agent.pd)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 396, in _process_internal_ports
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     self.internal_network_added(p)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 328, in internal_network_added
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     INTERNAL_DEV_PREFIX)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 303, in _internal_network_added
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     prefix=prefix)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 252, in plug
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     bridge, namespace, prefix)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 483, in plug_new
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     namespace2=namespace)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 144, in add_veth
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     self.ensure_namespace(namespace2)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 163, in ensure_namespace
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     ip = self.netns.add(name)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 793, in add
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     self._as_root([], ('add', name), use_root_namespace=True)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 280, in _as_root
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     use_root_namespace=use_root_namespace)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 80, in _as_root
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 89, in _execute
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     log_fail_as_error=log_fail_as_error)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 159, in execute
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info     raise RuntimeError(m)
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info RuntimeError:
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a']
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Exit code: 1
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Stdin:
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Stdout:
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info
  2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'ec62eace-0415-49b5-9c26-dca1677ba85a'
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 498, in _process_router_update
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 443, in _process_router_if_compatible
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 457, in _process_updated_router
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     ri.process(self)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 359, in call
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     self.logger(e)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 356, in call
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 692, in process
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     self._process_internal_ports(agent.pd)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 396, in _process_internal_ports
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     self.internal_network_added(p)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 328, in internal_network_added
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     INTERNAL_DEV_PREFIX)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 303, in _internal_network_added
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     prefix=prefix)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 252, in plug
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     bridge, namespace, prefix)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 483, in plug_new
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     namespace2=namespace)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 144, in add_veth
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     self.ensure_namespace(namespace2)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 163, in ensure_namespace
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     ip = self.netns.add(name)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 793, in add
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     self._as_root([], ('add', name), use_root_namespace=True)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 280, in _as_root
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     use_root_namespace=use_root_namespace)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 80, in _as_root
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     log_fail_as_error=self.log_fail_as_error)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 89, in _execute
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     log_fail_as_error=log_fail_as_error)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent   File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 159, in execute
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent     raise RuntimeError(m)
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent RuntimeError:
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a']
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Exit code: 1
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Stdin:
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Stdout:
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent
  2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1528977/+subscriptions
References

[Bug 1528977] [NEW] Neutron router does not work with latest iproute package included in CentOS-7.2-1511
From: Andrew Poltavchenko, 2015-12-24