yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1786047] [NEW] neutron-dhcp-agent is unable to set network namespaces

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dilip Renkila <dilip.renkila278@xxxxxxxxx>
Date: Wed, 08 Aug 2018 14:14:57 -0000
Reply-to: Bug 1786047 <1786047@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi all,

I am running neutron-dhcp-agents on all my controller nodes. I figured
out that neutron-dhcp-agent has a problem to work with namespaces. It is
failing continuously. I have deployed openstack by using openstack
ansible and all my networking agents run on the hosts not on containers.


root@ctrl1:~# uname -a
Linux ctrl1 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
You have new mail in /var/mail/root


root@ctrl1:~# /openstack/venvs/neutron-18.0.0.0b3/bin/neutron-dhcp-agent --version
neutron-dhcp-agent 12.0.0.0rc2.dev241

root@ctrl1:~# ls -la /var/run/netns
total 0
drwxrwxrwx  2 root root  100 aug  8 11:59 .
drwxr-xr-x 44 root root 1520 aug  8 16:11 ..
----------  1 root root    0 aug  8 11:59 qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127
----------  1 root root    0 aug  7 11:28 qrouter-3755419c-2a20-4ed0-9012-af24863f2447
----------  1 root root    0 aug  7 11:28 snat-3755419c-2a20-4ed0-9012-af24863f2447

root@ctrl3:~# ip netns
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127
RTNETLINK answers: Invalid argument
qrouter-3755419c-2a20-4ed0-9012-af24863f2447
RTNETLINK answers: Invalid argument
snat-3755419c-2a20-4ed0-9012-af24863f2447


root@ctrl1:~# tail -f /var/log/neutron/neutron-dhcp-agent.log

2018-08-08 16:06:32.408 1039014 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
setting the network namespace "qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127" failed: Invalid argument

2018-08-08 16:06:32.409 1039014 WARNING neutron.agent.linux.ip_lib [-] Setting ['sysctl', '-w', 'net.ipv6.conf.default.accept_ra=0'] in namespace qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127 failed: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
setting the network namespace "qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127" failed: Invalid argument
.: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for b43d85c9-575d-40f5-8960-2ba26620e127.: OSError: [Errno 13] Permission denied
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 144, in call_driver
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     getattr(driver, action)(**action_kwargs)
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 217, in enable
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     interface_name = self.device_manager.setup(self.network)
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1438, in setup
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     namespace=network.namespace):
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1111, in ensure_device_is_ready
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     if not dev.link.address:
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 545, in address
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     return self.attributes.get('link/ether')
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 574, in attributes
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     self._parent.namespace)
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     return self.channel.remote_call(name, args, kwargs)
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     raise exc_type(*result[2])
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent OSError: [Errno 13] Permission denied
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent 
2018-08-08 16:06:32.420 1039014 INFO neutron.agent.dhcp.agent [-] Finished network b43d85c9-575d-40f5-8960-2ba26620e127 dhcp configuration

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: dhcp-agent neutron

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1786047

Title:
  neutron-dhcp-agent is unable to set network namespaces

Status in neutron:
  New

Bug description:
  Hi all,

  I am running neutron-dhcp-agents on all my controller nodes. I figured
  out that neutron-dhcp-agent has a problem to work with namespaces. It
  is failing continuously. I have deployed openstack by using openstack
  ansible and all my networking agents run on the hosts not on
  containers.

  
  root@ctrl1:~# uname -a
  Linux ctrl1 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  You have new mail in /var/mail/root

  
  root@ctrl1:~# /openstack/venvs/neutron-18.0.0.0b3/bin/neutron-dhcp-agent --version
  neutron-dhcp-agent 12.0.0.0rc2.dev241

  root@ctrl1:~# ls -la /var/run/netns
  total 0
  drwxrwxrwx  2 root root  100 aug  8 11:59 .
  drwxr-xr-x 44 root root 1520 aug  8 16:11 ..
  ----------  1 root root    0 aug  8 11:59 qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127
  ----------  1 root root    0 aug  7 11:28 qrouter-3755419c-2a20-4ed0-9012-af24863f2447
  ----------  1 root root    0 aug  7 11:28 snat-3755419c-2a20-4ed0-9012-af24863f2447

  root@ctrl3:~# ip netns
  RTNETLINK answers: Invalid argument
  RTNETLINK answers: Invalid argument
  RTNETLINK answers: Invalid argument
  RTNETLINK answers: Invalid argument
  qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127
  RTNETLINK answers: Invalid argument
  qrouter-3755419c-2a20-4ed0-9012-af24863f2447
  RTNETLINK answers: Invalid argument
  snat-3755419c-2a20-4ed0-9012-af24863f2447


  root@ctrl1:~# tail -f /var/log/neutron/neutron-dhcp-agent.log

  2018-08-08 16:06:32.408 1039014 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  setting the network namespace "qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127" failed: Invalid argument

  2018-08-08 16:06:32.409 1039014 WARNING neutron.agent.linux.ip_lib [-] Setting ['sysctl', '-w', 'net.ipv6.conf.default.accept_ra=0'] in namespace qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127 failed: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  setting the network namespace "qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127" failed: Invalid argument
  .: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for b43d85c9-575d-40f5-8960-2ba26620e127.: OSError: [Errno 13] Permission denied
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 144, in call_driver
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     getattr(driver, action)(**action_kwargs)
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 217, in enable
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     interface_name = self.device_manager.setup(self.network)
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1438, in setup
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     namespace=network.namespace):
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1111, in ensure_device_is_ready
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     if not dev.link.address:
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 545, in address
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     return self.attributes.get('link/ether')
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 574, in attributes
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     self._parent.namespace)
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     return self.channel.remote_call(name, args, kwargs)
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent   File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent     raise exc_type(*result[2])
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent OSError: [Errno 13] Permission denied
  2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent 
  2018-08-08 16:06:32.420 1039014 INFO neutron.agent.dhcp.agent [-] Finished network b43d85c9-575d-40f5-8960-2ba26620e127 dhcp configuration

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1786047/+subscriptions
Follow ups

[Bug 1786047] Re: neutron-dhcp-agent is unable to set network namespaces
From: Launchpad Bug Tracker, 2018-10-09