yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1892362] [NEW] Restarting L3 agent when PD is used fails due to IPAddressAlreadyExists error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Slawek Kaplonski <1892362@xxxxxxxxxxxxxxxxxx>
Date: Thu, 20 Aug 2020 13:12:25 -0000
Reply-to: Bug 1892362 <1892362@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When there is router with subnet with enabled prefix delegation plugged
and L3 agent is restarted it's failing with error like:

2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 7ef0983f-60a4-4389-909c-785eb6c9a51e: neutron.privileged.agent.linux.ip_lib.IpAddressAlreadyExists: IP address fe80::f816:3eff:fec8:b9a8 already configured on qg-351b89d8-61.
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 694, in _process_routers_if_compatible
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 540, in _process_router_if_compatible
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 581, in _process_updated_router
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     ri.process()
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 161, in call
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self.logger(e)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self.force_reraise()
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     raise value
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 158, in call
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1191, in process
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self.process_external()
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 949, in process_external
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 833, in _process_external_gateway
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     interface_name)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     return f(*args, **kwargs)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/pd.py", line 144, in add_gw_interface
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self._add_lla(router, bind_lla_with_mask)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/pd.py", line 220, in _add_lla
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     'link')
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/interface.py", line 208, in add_ipv6_addr
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     device.addr.add(str(net), scope)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 508, in add
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     add_broadcast)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 872, in add_ip_address
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     device, namespace, scope, broadcast)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     return input_func(*args, **kwargs)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 244, in _wrap
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     return self.channel.remote_call(name, args, kwargs)
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 203, in remote_call
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     raise exc_type(*result[2])
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.IpAddressAlreadyExists: IP address fe80::f816:3eff:fec8:b9a8 already configured on qg-351b89d8-61.
2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent

To workaround this problem I had to e.g. remove qrouter-XXX namespace
from the node and then restart L3 agent. In that case all was configured
without this error.

** Affects: neutron
     Importance: Medium
         Status: Confirmed


** Tags: l3-dvr-backlog

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1892362

Title:
  Restarting L3 agent when PD is used fails due to
  IPAddressAlreadyExists error

Status in neutron:
  Confirmed

Bug description:
  When there is router with subnet with enabled prefix delegation
  plugged and L3 agent is restarted it's failing with error like:

  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 7ef0983f-60a4-4389-909c-785eb6c9a51e: neutron.privileged.agent.linux.ip_lib.IpAddressAlreadyExists: IP address fe80::f816:3eff:fec8:b9a8 already configured on qg-351b89d8-61.
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent Traceback (most recent call last):
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 694, in _process_routers_if_compatible
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self._process_router_if_compatible(router)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 540, in _process_router_if_compatible
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self._process_updated_router(router)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/agent.py", line 581, in _process_updated_router
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     ri.process()
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 161, in call
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self.logger(e)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self.force_reraise()
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     six.reraise(self.type_, self.value, self.tb)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     raise value
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/common/utils.py", line 158, in call
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     return func(*args, **kwargs)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 1191, in process
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self.process_external()
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 949, in process_external
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self._process_external_gateway(ex_gw_port)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/l3/router_info.py", line 833, in _process_external_gateway
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     interface_name)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     return f(*args, **kwargs)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/pd.py", line 144, in add_gw_interface
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     self._add_lla(router, bind_lla_with_mask)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/pd.py", line 220, in _add_lla
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     'link')
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/interface.py", line 208, in add_ipv6_addr
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     device.addr.add(str(net), scope)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 508, in add
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     add_broadcast)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 872, in add_ip_address
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     device, namespace, scope, broadcast)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     return input_func(*args, **kwargs)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 244, in _wrap
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     return self.channel.remote_call(name, args, kwargs)
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent   File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 203, in remote_call
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent     raise exc_type(*result[2])
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.IpAddressAlreadyExists: IP address fe80::f816:3eff:fec8:b9a8 already configured on qg-351b89d8-61.
  2020-08-20 12:54:33.390 487489 ERROR neutron.agent.l3.agent

  To workaround this problem I had to e.g. remove qrouter-XXX namespace
  from the node and then restart L3 agent. In that case all was
  configured without this error.

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1892362/+subscriptions
Follow ups

[Bug 1892362] Re: Restarting L3 agent when PD is used fails due to IPAddressAlreadyExists error
From: OpenStack Infra, 2020-08-25