yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1728642] [NEW] corrupted namespace blasted ovs bridge with thousands of dangling port

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yong sheng gong <gong_ys2004@xxxxxxxxxx>
Date: Mon, 30 Oct 2017 16:04:48 -0000
Reply-to: Bug 1728642 <1728642@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

when dhcp namespace is corrupted somehow, ovs bridge will be blasted
with thousands of dangling ports which are created by dhcp agent.

the corrupted namespace will cause following exception:

2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp [req-db1e4f25-2263-49e9-ba5b-308ea9ccfdec - - - - -] Unable to plug DHCP port for network 0c59667a-433a-4e97-9568-07ee6210c98b. Releasing port.
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp Traceback (most recent call last):
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1407, in setup
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     self.plug(network, port, interface_name)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1375, in plug
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     mtu=network.get('mtu'))
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 268, in plug
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     bridge, namespace, prefix, mtu)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 389, in plug_new
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     namespace_obj.add_device_to_namespace(ns_dev)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 232, in add_device_to_namespace
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     device.link.set_netns(self.namespace)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 516, in set_netns
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     self._as_root([], ('set', self.name, 'netns', namespace))
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 364, in _as_root
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     use_root_namespace=use_root_namespace)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 100, in _as_root
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     log_fail_as_error=self.log_fail_as_error)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 109, in _execute
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     log_fail_as_error=log_fail_as_error)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 156, in execute
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     raise ProcessExecutionError(msg, returncode=returncode)
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp
2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp
2017-10-30 14:12:35.479 6 ERROR neutron.agent.linux.utils [req-29d446ad-eed5-47a0-bfc7-496dad2d35f2 - - - - -] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument

** Affects: neutron
     Importance: Undecided
     Assignee: yong sheng gong (gongysh)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1728642

Title:
  corrupted namespace blasted ovs bridge with thousands of dangling port

Status in neutron:
  In Progress

Bug description:
  when dhcp namespace is corrupted somehow, ovs bridge will be blasted
  with thousands of dangling ports which are created by dhcp agent.

  the corrupted namespace will cause following exception:

  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp [req-db1e4f25-2263-49e9-ba5b-308ea9ccfdec - - - - -] Unable to plug DHCP port for network 0c59667a-433a-4e97-9568-07ee6210c98b. Releasing port.
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp Traceback (most recent call last):
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1407, in setup
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     self.plug(network, port, interface_name)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1375, in plug
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     mtu=network.get('mtu'))
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 268, in plug
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     bridge, namespace, prefix, mtu)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 389, in plug_new
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     namespace_obj.add_device_to_namespace(ns_dev)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 232, in add_device_to_namespace
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     device.link.set_netns(self.namespace)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 516, in set_netns
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     self._as_root([], ('set', self.name, 'netns', namespace))
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 364, in _as_root
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     use_root_namespace=use_root_namespace)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 100, in _as_root
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     log_fail_as_error=self.log_fail_as_error)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 109, in _execute
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     log_fail_as_error=log_fail_as_error)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp   File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 156, in execute
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp     raise ProcessExecutionError(msg, returncode=returncode)
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp ProcessExecutionError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp
  2017-10-30 14:12:35.347 6 ERROR neutron.agent.linux.dhcp
  2017-10-30 14:12:35.479 6 ERROR neutron.agent.linux.utils [req-29d446ad-eed5-47a0-bfc7-496dad2d35f2 - - - - -] Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1728642/+subscriptions
Follow ups

[Bug 1728642] Re: corrupted namespace blasted ovs bridge with thousands of dangling port
From: OpenStack Infra, 2018-09-05