yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1953165] [NEW] DHCP agent fails to fully configure DHCP namespaces because of duplicate address detected

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Pierre Riteau <1953165@xxxxxxxxxxxxxxxxxx>
Date: Fri, 03 Dec 2021 14:00:00 -0000
Reply-to: Bug 1953165 <1953165@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

After upgrading a Neutron/ML2 OVS deployment from Ussuri to Victoria,
updating the host OS from CentOS Linux 8 to CentOS Stream 8, and
rebooting, DHCP was not functional on some but not all networks.

DHCP agent logs included the following error multiple times:

2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent [-] Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected: neutron.agent.linux.ip_lib.AddressNotReady: Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 165, in call
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     return func(*args, **kwargs)
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 401, in safe_configure_dhcp_for_network
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     self.configure_dhcp_for_network(network)
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     result = f(*args, **kwargs)
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 415, in configure_dhcp_for_network
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     self.update_isolated_metadata_proxy(network)
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     result = f(*args, **kwargs)
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 758, in update_isolated_metadata_proxy
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     self.enable_isolated_metadata_proxy(network)
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     result = f(*args, **kwargs)
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 816, in enable_isolated_metadata_proxy
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     self.conf, bind_address=constants.METADATA_V4_IP, **kwargs)
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/metadata/driver.py", line 271, in spawn_monitored_metadata_proxy
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     ).wait_until_address_ready(address=bind_address_v6)
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 597, in wait_until_address_ready
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     exception=AddressNotReady(address=address, reason=errmsg))
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 701, in wait_until_true
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     while not predicate():
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 591, in is_address_ready
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     address=address, reason=_('Duplicate address detected'))
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent neutron.agent.linux.ip_lib.AddressNotReady: Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected
2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent 

The tap interface inside each affected qdhcp namespace was in a state
like this:

35: tap0f8bb343-c1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:ed:6f:60 brd ff:ff:ff:ff:ff:ff
    inet 169.254.169.254/32 brd 169.254.169.254 scope global tap0f8bb343-c1
       valid_lft forever preferred_lft forever
    inet 10.18.0.10/16 brd 10.18.255.255 scope global tap0f8bb343-c1
       valid_lft forever preferred_lft forever
    inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative 
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:feed:6f60/64 scope link 
       valid_lft forever preferred_lft forever

Note the dadfailed status on the fe80::a9fe:a9fe/64 address, which
caused Neutron to raise an AddressNotReady exception.

I tried restarting dhcp-agent multiple times. Occasionally DHCP for one
network would configure correctly, but most of the times the list of
affected networks would stay the same.

I found out that removing the fe80::a9fe:a9fe/64 address from the tap
interface of each affected namespace followed by restarting dhcp-agent
fixed the issue: no more dadfailed status.

Version information:

* OpenStack Victoria deployed with Kolla source images
* neutron 17.2.2.dev70 (using stable/victoria from 2021-11-28)
* CentOS Stream release 8
* Linux kernel 4.18.0-348.2.1.el8_5.x86_64 #1 SMP Tue Nov 16 14:42:35 UTC 2021

** Affects: neutron
     Importance: Undecided
         Status: New

** Summary changed:

- DHCP agent fails to configure DHCP namespaces because of duplicate address detected
+ DHCP agent fails to fully configure DHCP namespaces because of duplicate address detected

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1953165

Title:
  DHCP agent fails to fully configure DHCP namespaces because of
  duplicate address detected

Status in neutron:
  New

Bug description:
  After upgrading a Neutron/ML2 OVS deployment from Ussuri to Victoria,
  updating the host OS from CentOS Linux 8 to CentOS Stream 8, and
  rebooting, DHCP was not functional on some but not all networks.

  DHCP agent logs included the following error multiple times:

  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent [-] Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected: neutron.agent.linux.ip_lib.AddressNotReady: Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 165, in call
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     return func(*args, **kwargs)
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 401, in safe_configure_dhcp_for_network
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     self.configure_dhcp_for_network(network)
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     result = f(*args, **kwargs)
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 415, in configure_dhcp_for_network
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     self.update_isolated_metadata_proxy(network)
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     result = f(*args, **kwargs)
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 758, in update_isolated_metadata_proxy
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     self.enable_isolated_metadata_proxy(network)
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/osprofiler/profiler.py", line 160, in wrapper
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     result = f(*args, **kwargs)
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/dhcp/agent.py", line 816, in enable_isolated_metadata_proxy
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     self.conf, bind_address=constants.METADATA_V4_IP, **kwargs)
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/metadata/driver.py", line 271, in spawn_monitored_metadata_proxy
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     ).wait_until_address_ready(address=bind_address_v6)
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 597, in wait_until_address_ready
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     exception=AddressNotReady(address=address, reason=errmsg))
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/common/utils.py", line 701, in wait_until_true
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     while not predicate():
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent   File "/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/agent/linux/ip_lib.py", line 591, in is_address_ready
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent     address=address, reason=_('Duplicate address detected'))
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent neutron.agent.linux.ip_lib.AddressNotReady: Failure waiting for address fe80::a9fe:a9fe to become ready: Duplicate address detected
  2021-11-30 17:05:35.475 7 ERROR neutron.agent.dhcp.agent 

  The tap interface inside each affected qdhcp namespace was in a state
  like this:

  35: tap0f8bb343-c1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default qlen 1000
      link/ether fa:16:3e:ed:6f:60 brd ff:ff:ff:ff:ff:ff
      inet 169.254.169.254/32 brd 169.254.169.254 scope global tap0f8bb343-c1
         valid_lft forever preferred_lft forever
      inet 10.18.0.10/16 brd 10.18.255.255 scope global tap0f8bb343-c1
         valid_lft forever preferred_lft forever
      inet6 fe80::a9fe:a9fe/64 scope link dadfailed tentative 
         valid_lft forever preferred_lft forever
      inet6 fe80::f816:3eff:feed:6f60/64 scope link 
         valid_lft forever preferred_lft forever

  Note the dadfailed status on the fe80::a9fe:a9fe/64 address, which
  caused Neutron to raise an AddressNotReady exception.

  I tried restarting dhcp-agent multiple times. Occasionally DHCP for
  one network would configure correctly, but most of the times the list
  of affected networks would stay the same.

  I found out that removing the fe80::a9fe:a9fe/64 address from the tap
  interface of each affected namespace followed by restarting dhcp-agent
  fixed the issue: no more dadfailed status.

  Version information:

  * OpenStack Victoria deployed with Kolla source images
  * neutron 17.2.2.dev70 (using stable/victoria from 2021-11-28)
  * CentOS Stream release 8
  * Linux kernel 4.18.0-348.2.1.el8_5.x86_64 #1 SMP Tue Nov 16 14:42:35 UTC 2021

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1953165/+subscriptions
Follow ups

[Bug 1953165] Re: DHCP agent fails to fully configure DHCP namespaces because of duplicate address detected
From: Rodolfo Alonso, 2023-07-10
[Bug 1953165] Re: DHCP agent fails to fully configure DHCP namespaces because of duplicate address detected
From: Maximilian Stinsky, 2022-11-25