yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1561695] [NEW] neutron-dhcp-agent generates thousands of interfaces on a failure

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sam Yaple <sam@xxxxxxxxx>
Date: Thu, 24 Mar 2016 18:51:41 -0000
Reply-to: Bug 1561695 <1561695@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I ran into slowness on a new deploy of mitaka-rc1 code with neutron. I
had ~13,000 tap devices that were created by dhcp-agent. The neutron
database does not have these ports. As far as I can tell, neutron is no
longer aware, or cares about those ports but they remain on the node
(and in OpenVSwitch so a reboot wouldnt clear them).

I do not know how the initial failure happened, but to reproduce this
you can do the following:

1. Stop dhcp agent (and anything using the network namespace).
2. ip netns del qdhcp-8e5d7a66-df5d-4e36-8446-3c2148e53f02
3. touch /run/netns/qdhcp-8e5d7a66-df5d-4e36-8446-3c2148e53f02
4 Start the dhcp agent and watch it continually try to create (and then fail to cleanup) tap interfaces

Over the course of ~4 hours this issue generate 13,000 interfaces and
4GB of logs (debug was turned on). How the initial issue came about I do
not know but it did happen in normal usage. I believe the proper fix
here would be _always_ clean up tap devices even on failures but I am
not familiar with the neutron code enough to fix this.

The output of `ip netns` when it has an invalid namespace looks like
this:

# ip netns
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
qdhcp-8e5d7a66-df5d-4e36-8446-3c2148e53f02

The stack trace in neutron-dhcp-agent is:

2016-03-24 18:42:12.165 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=ofport', 'list', 'Interface', 'tap42983a07-e0'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
2016-03-24 18:42:12.275 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
2016-03-24 18:42:12.276 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'tap42983a07-e0', 'address', 'fa:16:3e:79:1b:0a'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
2016-03-24 18:42:12.384 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
2016-03-24 18:42:12.385 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'tap42983a07-e0', 'mtu', '9000'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
2016-03-24 18:42:12.495 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
2016-03-24 18:42:12.496 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', '-o', 'netns', 'list'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
2016-03-24 18:42:12.604 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
2016-03-24 18:42:12.605 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'tap42983a07-e0', 'netns', 'qdhcp-8e5d7a66-df5d-4e36-8446-3c2148e53f02'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
2016-03-24 18:42:12.709 1 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument

2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp [-] Unable to plug DHCP port for network 8e5d7a66-df5d-4e36-8446-3c2148e53f02. Releasing port.
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp Traceback (most recent call last):
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1234, in setup
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     mtu=network.get('mtu'))
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 248, in plug
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     bridge, namespace, prefix, mtu)
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 346, in plug_new
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     namespace_obj.add_device_to_namespace(ns_dev)
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 216, in add_device_to_namespace
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     device.link.set_netns(self.namespace)
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 514, in set_netns
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     self._as_root([], ('set', self.name, 'netns', namespace))
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 365, in _as_root
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     use_root_namespace=use_root_namespace)
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     log_fail_as_error=self.log_fail_as_error)
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     log_fail_as_error=log_fail_as_error)
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     raise RuntimeError(msg)
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp 
2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp 
2016-03-24 18:42:12.711 1 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 559dc40172904849a6cda4efebd85c38 exchange 'neutron' topic 'q-plugin' _send /var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-03-24 18:42:12.858 1 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 559dc40172904849a6cda4efebd85c38 __call__ /var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:302
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 8e5d7a66-df5d-4e36-8446-3c2148e53f02.
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 112, in call_driver
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     getattr(driver, action)(**action_kwargs)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 208, in enable
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     interface_name = self.device_manager.setup(self.network)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1240, in setup
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     self.plugin.release_dhcp_port(network.id, port.device_id)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     self.force_reraise()
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     six.reraise(self.type_, self.value, self.tb)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1234, in setup
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     mtu=network.get('mtu'))
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 248, in plug
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     bridge, namespace, prefix, mtu)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 346, in plug_new
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     namespace_obj.add_device_to_namespace(ns_dev)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 216, in add_device_to_namespace
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     device.link.set_netns(self.namespace)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 514, in set_netns
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     self._as_root([], ('set', self.name, 'netns', namespace))
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 365, in _as_root
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     use_root_namespace=use_root_namespace)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     log_fail_as_error=self.log_fail_as_error)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     log_fail_as_error=log_fail_as_error)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     raise RuntimeError(msg)
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent 
2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent 
2016-03-24 18:42:12.859 1 INFO neutron.agent.dhcp.agent [-] Finished network 8e5d7a66-df5d-4e36-8446-3c2148e53f02 dhcp configuration
2016-03-24 18:42:12.859 1 INFO neutron.agent.dhcp.agent [-] Synchronizing state complete
2016-03-24 18:42:12.859 1 DEBUG oslo_concurrency.lockutils [-] Lock "dhcp-agent" released by "neutron.agent.dhcp.agent.sync_state" :: held 1.626s inner /var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1561695

Title:
  neutron-dhcp-agent generates thousands of interfaces on a failure

Status in neutron:
  New

Bug description:
  I ran into slowness on a new deploy of mitaka-rc1 code with neutron. I
  had ~13,000 tap devices that were created by dhcp-agent. The neutron
  database does not have these ports. As far as I can tell, neutron is
  no longer aware, or cares about those ports but they remain on the
  node (and in OpenVSwitch so a reboot wouldnt clear them).

  I do not know how the initial failure happened, but to reproduce this
  you can do the following:

  1. Stop dhcp agent (and anything using the network namespace).
  2. ip netns del qdhcp-8e5d7a66-df5d-4e36-8446-3c2148e53f02
  3. touch /run/netns/qdhcp-8e5d7a66-df5d-4e36-8446-3c2148e53f02
  4 Start the dhcp agent and watch it continually try to create (and then fail to cleanup) tap interfaces

  Over the course of ~4 hours this issue generate 13,000 interfaces and
  4GB of logs (debug was turned on). How the initial issue came about I
  do not know but it did happen in normal usage. I believe the proper
  fix here would be _always_ clean up tap devices even on failures but I
  am not familiar with the neutron code enough to fix this.

  The output of `ip netns` when it has an invalid namespace looks like
  this:

  # ip netns
  RTNETLINK answers: Invalid argument
  RTNETLINK answers: Invalid argument
  qdhcp-8e5d7a66-df5d-4e36-8446-3c2148e53f02

  The stack trace in neutron-dhcp-agent is:

  2016-03-24 18:42:12.165 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', '--columns=ofport', 'list', 'Interface', 'tap42983a07-e0'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
  2016-03-24 18:42:12.275 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
  2016-03-24 18:42:12.276 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'tap42983a07-e0', 'address', 'fa:16:3e:79:1b:0a'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
  2016-03-24 18:42:12.384 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
  2016-03-24 18:42:12.385 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'tap42983a07-e0', 'mtu', '9000'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
  2016-03-24 18:42:12.495 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
  2016-03-24 18:42:12.496 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', '-o', 'netns', 'list'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
  2016-03-24 18:42:12.604 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:142
  2016-03-24 18:42:12.605 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'link', 'set', 'tap42983a07-e0', 'netns', 'qdhcp-8e5d7a66-df5d-4e36-8446-3c2148e53f02'] create_process /var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py:84
  2016-03-24 18:42:12.709 1 ERROR neutron.agent.linux.utils [-] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument

  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp [-] Unable to plug DHCP port for network 8e5d7a66-df5d-4e36-8446-3c2148e53f02. Releasing port.
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp Traceback (most recent call last):
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1234, in setup
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     mtu=network.get('mtu'))
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 248, in plug
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     bridge, namespace, prefix, mtu)
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 346, in plug_new
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     namespace_obj.add_device_to_namespace(ns_dev)
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 216, in add_device_to_namespace
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     device.link.set_netns(self.namespace)
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 514, in set_netns
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     self._as_root([], ('set', self.name, 'netns', namespace))
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 365, in _as_root
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     use_root_namespace=use_root_namespace)
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     log_fail_as_error=self.log_fail_as_error)
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     log_fail_as_error=log_fail_as_error)
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp     raise RuntimeError(msg)
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp 
  2016-03-24 18:42:12.710 1 ERROR neutron.agent.linux.dhcp 
  2016-03-24 18:42:12.711 1 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 559dc40172904849a6cda4efebd85c38 exchange 'neutron' topic 'q-plugin' _send /var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:454
  2016-03-24 18:42:12.858 1 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 559dc40172904849a6cda4efebd85c38 __call__ /var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:302
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for 8e5d7a66-df5d-4e36-8446-3c2148e53f02.
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 112, in call_driver
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     getattr(driver, action)(**action_kwargs)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 208, in enable
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     interface_name = self.device_manager.setup(self.network)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1240, in setup
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     self.plugin.release_dhcp_port(network.id, port.device_id)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     self.force_reraise()
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     six.reraise(self.type_, self.value, self.tb)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1234, in setup
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     mtu=network.get('mtu'))
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 248, in plug
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     bridge, namespace, prefix, mtu)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 346, in plug_new
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     namespace_obj.add_device_to_namespace(ns_dev)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 216, in add_device_to_namespace
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     device.link.set_netns(self.namespace)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 514, in set_netns
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     self._as_root([], ('set', self.name, 'netns', namespace))
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 365, in _as_root
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     use_root_namespace=use_root_namespace)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 95, in _as_root
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     log_fail_as_error=self.log_fail_as_error)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 104, in _execute
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     log_fail_as_error=log_fail_as_error)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent     raise RuntimeError(msg)
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent 
  2016-03-24 18:42:12.859 1 ERROR neutron.agent.dhcp.agent 
  2016-03-24 18:42:12.859 1 INFO neutron.agent.dhcp.agent [-] Finished network 8e5d7a66-df5d-4e36-8446-3c2148e53f02 dhcp configuration
  2016-03-24 18:42:12.859 1 INFO neutron.agent.dhcp.agent [-] Synchronizing state complete
  2016-03-24 18:42:12.859 1 DEBUG oslo_concurrency.lockutils [-] Lock "dhcp-agent" released by "neutron.agent.dhcp.agent.sync_state" :: held 1.626s inner /var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1561695/+subscriptions
Follow ups

[Bug 1561695] Re: neutron-dhcp-agent generates thousands of interfaces on a failure
From: OpenStack Infra, 2017-09-09