yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1903835] [NEW] linuxbridge-agent ERRORS with pyroute2.netlink.exceptions.NetlinkError if ipv6 link local address exist

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jake Yip <1903835@xxxxxxxxxxxxxxxxxx>
Date: Wed, 11 Nov 2020 10:30:18 -0000
Reply-to: Bug 1903835 <1903835@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi all. We are running neutron train. After a minor version upgrade
recently, linuxbridge-agent started throwing errors like this:


2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-947d5add-cdd0-437d-a2eb-1c17d2556cd3 - - - - -] Error in agent loop. Devices info: {'current': {'tap25db63b7-9b', 'tapa6658716-0e', 'tapeae0251d-48', 'tapfeeae322-76', 'tap2dcf0e81-2d', 'tapd908490e-6f', 'tap4208ea46-9f', 'tape4a832ef
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     sync = self.process_network_devices(device_info)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = f(*args, **kwargs)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 214, in process_network_devices
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     resync_a = self.treat_devices_added_updated(devices_added_updated)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = f(*args, **kwargs)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 231, in treat_devices_added_updated
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self._process_device_if_exists(device_details)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 258, in _process_device_if_exists
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     device, device_details['device_owner'])
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 588, in plug_interface
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     network_segment.mtu)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 524, in add_tap_interface
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return False
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.force_reraise()
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     six.reraise(self.type_, self.value, self.tb)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     raise value
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 516, in add_tap_interface
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     tap_device_name, device_owner, mtu)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 549, in _add_tap_interface
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     mtu):
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 500, in ensure_physical_in_bridge
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     physical_interface)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 288, in ensure_flat_bridge
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     if self.ensure_bridge(bridge_name, physical_interface):
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 458, in ensure_bridge
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.update_interface_ip_details(bridge_name, interface)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 420, in update_interface_ip_details
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     gateway)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 396, in _update_interface_ip_details
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     dst_device.addr.add(cidr=ip['cidr'])
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/agent/linux/ip_lib.py", line 528, in add
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     add_broadcast)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/agent/linux/ip_lib.py", line 802, in add_ip_address
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     device, namespace, scope, broadcast)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/privileged/agent/linux/ip_lib.py", line 73, in sync_inner
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return input_func(*args, **kwargs)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 245, in _wrap
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return self.channel.remote_call(name, args, kwargs)
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in remote_call
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     raise exc_type(*result[2])
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent pyroute2.netlink.exceptions.NetlinkError: (13, 'Permission denied')
2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent 


I think I've traced it down to the (backported) change https://review.opendev.org/#/c/747627/ 

It happens because the linuxbridge-agent interface has a link-local ipv6
address. Prior to this patch, linuxbridge-agent works fine. With this
patch, it errors. Removing the link-local ipv6 address will solve the
problem.

It may be the correct way is to not have ipv6 link-local addresses on
linuxbridge interface; we are working towards removing them. Just
thought that I might put this up as a bug, in case operators and
developers run into the same problem.

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1903835

Title:
  linuxbridge-agent ERRORS with pyroute2.netlink.exceptions.NetlinkError
  if ipv6 link local address exist

Status in neutron:
  New

Bug description:
  Hi all. We are running neutron train. After a minor version upgrade
  recently, linuxbridge-agent started throwing errors like this:

  
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent [req-947d5add-cdd0-437d-a2eb-1c17d2556cd3 - - - - -] Error in agent loop. Devices info: {'current': {'tap25db63b7-9b', 'tapa6658716-0e', 'tapeae0251d-48', 'tapfeeae322-76', 'tap2dcf0e81-2d', 'tapd908490e-6f', 'tap4208ea46-9f', 'tape4a832ef
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent Traceback (most recent call last):
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 465, in daemon_loop
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     sync = self.process_network_devices(device_info)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = f(*args, **kwargs)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 214, in process_network_devices
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     resync_a = self.treat_devices_added_updated(devices_added_updated)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     result = f(*args, **kwargs)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 231, in treat_devices_added_updated
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self._process_device_if_exists(device_details)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/agent/_common_agent.py", line 258, in _process_device_if_exists
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     device, device_details['device_owner'])
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 588, in plug_interface
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     network_segment.mtu)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 524, in add_tap_interface
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return False
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.force_reraise()
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     six.reraise(self.type_, self.value, self.tb)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/six.py", line 693, in reraise
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     raise value
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 516, in add_tap_interface
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     tap_device_name, device_owner, mtu)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 549, in _add_tap_interface
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     mtu):
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 500, in ensure_physical_in_bridge
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     physical_interface)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 288, in ensure_flat_bridge
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     if self.ensure_bridge(bridge_name, physical_interface):
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 458, in ensure_bridge
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     self.update_interface_ip_details(bridge_name, interface)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 420, in update_interface_ip_details
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     gateway)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py", line 396, in _update_interface_ip_details
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     dst_device.addr.add(cidr=ip['cidr'])
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/agent/linux/ip_lib.py", line 528, in add
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     add_broadcast)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/agent/linux/ip_lib.py", line 802, in add_ip_address
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     device, namespace, scope, broadcast)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/neutron/neutron/privileged/agent/linux/ip_lib.py", line 73, in sync_inner
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return input_func(*args, **kwargs)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 245, in _wrap
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     return self.channel.remote_call(name, args, kwargs)
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent   File "/opt/train/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in remote_call
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent     raise exc_type(*result[2])
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent pyroute2.netlink.exceptions.NetlinkError: (13, 'Permission denied')
  2020-11-11 21:13:50.524 6088 ERROR neutron.plugins.ml2.drivers.agent._common_agent 

  
  I think I've traced it down to the (backported) change https://review.opendev.org/#/c/747627/ 

  It happens because the linuxbridge-agent interface has a link-local
  ipv6 address. Prior to this patch, linuxbridge-agent works fine. With
  this patch, it errors. Removing the link-local ipv6 address will solve
  the problem.

  It may be the correct way is to not have ipv6 link-local addresses on
  linuxbridge interface; we are working towards removing them. Just
  thought that I might put this up as a bug, in case operators and
  developers run into the same problem.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1903835/+subscriptions
Follow ups

[Bug 1903835] Re: linuxbridge-agent ERRORS with pyroute2.netlink.exceptions.NetlinkError if ipv6 link local address exist
From: Slawek Kaplonski, 2020-11-17
[Bug 1903835] Re: linuxbridge-agent ERRORS with pyroute2.netlink.exceptions.NetlinkError if ipv6 link local address exist
From: Rodolfo Alonso, 2020-11-16