yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1640202] [NEW] L3 agent failure to setup floating IPs

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Nicolas Vila <nico@xxxxxxxxxxx>
Date: Tue, 08 Nov 2016 15:13:58 -0000
Reply-to: Bug 1640202 <1640202@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hello,

We have a Mitaka deploy with several projects running, with 3 controller
nodes and 4 compute nodes. After restarting the L3 agent, for one
specific tenant the floating IPs that are attached to an instance are
not actually attached, L3 agent is throwing the following error:

2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info [-] L3 agent failure to setup floating IPs
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 342, in configure_fip_addresses
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     return self.process_floating_ip_addresses(interface_name)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 314, in process_floating_ip_addresses
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     fip, interface_name, device)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 157, in add_floating_ip
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     self.floating_ip_added_dist(fip, ip_cidr)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 88, in floating_ip_added_dist
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     device.route.add_route(fip_cidr, str(rtr_2_fip.ip))
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 841, in add_route
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     self._run_as_root_detect_device_not_found([ip_version], tuple(args))
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 699, in _run_as_root_detect_device_not_found
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     raise exceptions.DeviceNotFoundError(device_name=self.name)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     self.force_reraise()
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     six.reraise(self.type_, self.value, self.tb)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 694, in _run_as_root_detect_device_not_found
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     return self._as_root(*args, **kwargs)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 361, in _as_root
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     use_root_namespace=use_root_namespace)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 103, in _execute
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     log_fail_as_error=log_fail_as_error)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     raise RuntimeError(msg)
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Network is unreachable
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info 
2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info 
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info [-] Failed to process floating IPs.
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 815, in process_external
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info     fip_statuses = self.configure_fip_addresses(interface_name)
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 347, in configure_fip_addresses
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info     raise n_exc.FloatingIpSetupException(msg)
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info FloatingIpSetupException: L3 agent failure to setup floating IPs
2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info 

As of now, every time we try to attach/detach the floating IP on an
instance the L3 agent shows the same error. Neutron package versions
are:

ii  neutron-common                              2:8.2.0-0ubuntu1~cloud0
ii  neutron-dhcp-agent                          2:8.2.0-0ubuntu1~cloud0
ii  neutron-l3-agent                            2:8.2.0-0ubuntu1~cloud0
ii  neutron-metadata-agent                      2:8.2.0-0ubuntu1~cloud0
ii  neutron-openvswitch-agent                   2:8.2.0-0ubuntu1~cloud0
ii  neutron-plugin-ml2                          2:8.2.0-0ubuntu1~cloud0

OS: Ubuntu 14.04, uname: Linux compute01 3.16.0-77-generic
#99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64 x86_64
GNU/Linux

** Affects: neutron
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1640202

Title:
  L3 agent failure to setup floating IPs

Status in neutron:
  New

Bug description:
  Hello,

  We have a Mitaka deploy with several projects running, with 3
  controller nodes and 4 compute nodes. After restarting the L3 agent,
  for one specific tenant the floating IPs that are attached to an
  instance are not actually attached, L3 agent is throwing the following
  error:

  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info [-] L3 agent failure to setup floating IPs
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 342, in configure_fip_addresses
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     return self.process_floating_ip_addresses(interface_name)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 314, in process_floating_ip_addresses
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     fip, interface_name, device)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 157, in add_floating_ip
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     self.floating_ip_added_dist(fip, ip_cidr)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/dvr_local_router.py", line 88, in floating_ip_added_dist
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     device.route.add_route(fip_cidr, str(rtr_2_fip.ip))
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 841, in add_route
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     self._run_as_root_detect_device_not_found([ip_version], tuple(args))
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 699, in _run_as_root_detect_device_not_found
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     raise exceptions.DeviceNotFoundError(device_name=self.name)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     self.force_reraise()
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     six.reraise(self.type_, self.value, self.tb)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 694, in _run_as_root_detect_device_not_found
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     return self._as_root(*args, **kwargs)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 361, in _as_root
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     use_root_namespace=use_root_namespace)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 94, in _as_root
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     log_fail_as_error=self.log_fail_as_error)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/ip_lib.py", line 103, in _execute
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     log_fail_as_error=log_fail_as_error)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py", line 140, in execute
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info     raise RuntimeError(msg)
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 2; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Network is unreachable
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info 
  2016-11-08 08:56:02.061 12466 ERROR neutron.agent.l3.router_info 
  2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info [-] Failed to process floating IPs.
  2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
  2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 815, in process_external
  2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info     fip_statuses = self.configure_fip_addresses(interface_name)
  2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info   File "/usr/lib/python2.7/dist-packages/neutron/agent/l3/router_info.py", line 347, in configure_fip_addresses
  2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info     raise n_exc.FloatingIpSetupException(msg)
  2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info FloatingIpSetupException: L3 agent failure to setup floating IPs
  2016-11-08 08:56:02.063 12466 ERROR neutron.agent.l3.router_info 

  As of now, every time we try to attach/detach the floating IP on an
  instance the L3 agent shows the same error. Neutron package versions
  are:

  ii  neutron-common                              2:8.2.0-0ubuntu1~cloud0
  ii  neutron-dhcp-agent                          2:8.2.0-0ubuntu1~cloud0
  ii  neutron-l3-agent                            2:8.2.0-0ubuntu1~cloud0
  ii  neutron-metadata-agent                      2:8.2.0-0ubuntu1~cloud0
  ii  neutron-openvswitch-agent                   2:8.2.0-0ubuntu1~cloud0
  ii  neutron-plugin-ml2                          2:8.2.0-0ubuntu1~cloud0

  OS: Ubuntu 14.04, uname: Linux compute01 3.16.0-77-generic
  #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10 UTC 2016 x86_64 x86_64
  x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1640202/+subscriptions
Follow ups

[Bug 1640202] Re: L3 agent failure to setup floating IPs
From: Rodolfo Alonso, 2022-12-16