yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2113923] [NEW] Neutron metadata agent fails to register in Epoxy

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alex Welsh <2113923@xxxxxxxxxxxxxxxxxx>
Date: Wed, 11 Jun 2025 08:10:57 -0000
Reply-to: Bug 2113923 <2113923@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Neutron Metadata agent fails to register in the Epoxy release.

On a fresh deployment, it just never registers
When upgrading from Caracal, it gets marked as down because no heartbeats are sent after the upgrade.
I have not tested Dalmatian.

Even though heartbeats fail, it still works fine. It responds and
requests can be seen in the logs.

Tested in a Kayobe AIO (Kolla containers)

Tested on Rocky 9 and Ubuntu Noble, both are affected

OVN is unaffected

Appears to have affected Kolla-Ansible since mid-march:
https://review.opendev.org/c/openstack/kolla-ansible/+/944194

Severity is high. No functional issues but lots of alerts firing .

Logs
====

Neutron server error (last heartbeat is time of upgrade):

Agent healthcheck: found 1 dead agents out of 4:
                Type       Last heartbeat host
      Metadata agent  2025-06-09 13:29:05 alex-aio-epoxy-rocky06.novalocal

Openstack view after upgrade:
$ openstack network agent list -c "Agent Type" -c Alive -c State -c Binary
+--------------------+-------+-------+---------------------------+
| Agent Type         | Alive | State | Binary                    |
+--------------------+-------+-------+---------------------------+
| Open vSwitch agent | :-)   | UP    | neutron-openvswitch-agent |
| DHCP agent         | :-)   | UP    | neutron-dhcp-agent        |
| L3 agent           | :-)   | UP    | neutron-l3-agent          |
| Metadata agent     | XXX   | UP    | neutron-metadata-agent    |
+--------------------+-------+-------+---------------------------+

Neutron metadata agent response log:

2025-06-10 15:18:24.306 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Request: GET /latest/meta-data/public-ipv4/ HTTP/1.1
host: 169.254.169.254
user-agent: curl/7.84.0
accept: */*
x-neutron-network-id: 03739e2a-f552-43b1-9d6f-43db61d03aaf
x-forwarded-for: 10.0.0.59
connection: close

 handle /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:163
2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Acquiring lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:405
2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" acquired by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: waited 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:410
2025-06-10 15:18:24.311 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" "released" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: held 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:424
2025-06-10 15:18:24.833 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Got ports for remote_address 10.0.0.59, network_id 03739e2a-f552-43b1-9d6f-43db61d03aaf, remote_mac None, router_id None[{'id': 'cdbb6292-e301-4f5c-bc34-33e139eb97fb', 'name': '', 'network_id': '03739e2a-f552-43b1-9d6f-43db61d03aaf', 'tenant_id': '961e2027d2ac49d8b7e76065b42206ac', 'mac_address': 'fa:16:3e:d5:a3:a6', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'b68cc942-cfc2-4ee7-be20-5ce5e1f62353', 'device_owner': 'compute:nova', 'standard_attr_id': 662, 'fixed_ips': [{'subnet_id': 'c0cfe0fc-f3af-4ac1-9e11-a4fbbcd31167', 'ip_address': '10.0.0.59'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['f2f888b6-1209-4127-8981-acec08ec4eed'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'alex-aio-epoxy-rocky-7.novalocal', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system', 'bridge_name': 'br-int', 'bound_drivers': {'0': 'openvswitch'}}, 'port_security_enabled': True, 'tags': [], 'created_at': '2025-06-10T14:57:42Z', 'updated_at': '2025-06-10T14:57:47Z', 'revision_number': 5, 'project_id': '961e2027d2ac49d8b7e76065b42206ac'}] get_port /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:255

** Affects: neutron
     Importance: Undecided
         Status: New


** Tags: ovs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2113923

Title:
  Neutron metadata agent fails to register in Epoxy

Status in neutron:
  New

Bug description:
  Neutron Metadata agent fails to register in the Epoxy release.

  On a fresh deployment, it just never registers
  When upgrading from Caracal, it gets marked as down because no heartbeats are sent after the upgrade.
  I have not tested Dalmatian.

  Even though heartbeats fail, it still works fine. It responds and
  requests can be seen in the logs.

  Tested in a Kayobe AIO (Kolla containers)

  Tested on Rocky 9 and Ubuntu Noble, both are affected

  OVN is unaffected

  Appears to have affected Kolla-Ansible since mid-march:
  https://review.opendev.org/c/openstack/kolla-ansible/+/944194

  Severity is high. No functional issues but lots of alerts firing .

  Logs
  ====

  Neutron server error (last heartbeat is time of upgrade):

  Agent healthcheck: found 1 dead agents out of 4:
                  Type       Last heartbeat host
        Metadata agent  2025-06-09 13:29:05 alex-aio-epoxy-rocky06.novalocal

  Openstack view after upgrade:
  $ openstack network agent list -c "Agent Type" -c Alive -c State -c Binary
  +--------------------+-------+-------+---------------------------+
  | Agent Type         | Alive | State | Binary                    |
  +--------------------+-------+-------+---------------------------+
  | Open vSwitch agent | :-)   | UP    | neutron-openvswitch-agent |
  | DHCP agent         | :-)   | UP    | neutron-dhcp-agent        |
  | L3 agent           | :-)   | UP    | neutron-l3-agent          |
  | Metadata agent     | XXX   | UP    | neutron-metadata-agent    |
  +--------------------+-------+-------+---------------------------+

  Neutron metadata agent response log:

  2025-06-10 15:18:24.306 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Request: GET /latest/meta-data/public-ipv4/ HTTP/1.1
  host: 169.254.169.254
  user-agent: curl/7.84.0
  accept: */*
  x-neutron-network-id: 03739e2a-f552-43b1-9d6f-43db61d03aaf
  x-forwarded-for: 10.0.0.59
  connection: close

   handle /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:163
  2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Acquiring lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:405
  2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" acquired by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: waited 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:410
  2025-06-10 15:18:24.311 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" "released" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: held 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:424
  2025-06-10 15:18:24.833 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Got ports for remote_address 10.0.0.59, network_id 03739e2a-f552-43b1-9d6f-43db61d03aaf, remote_mac None, router_id None[{'id': 'cdbb6292-e301-4f5c-bc34-33e139eb97fb', 'name': '', 'network_id': '03739e2a-f552-43b1-9d6f-43db61d03aaf', 'tenant_id': '961e2027d2ac49d8b7e76065b42206ac', 'mac_address': 'fa:16:3e:d5:a3:a6', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'b68cc942-cfc2-4ee7-be20-5ce5e1f62353', 'device_owner': 'compute:nova', 'standard_attr_id': 662, 'fixed_ips': [{'subnet_id': 'c0cfe0fc-f3af-4ac1-9e11-a4fbbcd31167', 'ip_address': '10.0.0.59'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['f2f888b6-1209-4127-8981-acec08ec4eed'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'alex-aio-epoxy-rocky-7.novalocal', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system', 'bridge_name': 'br-int', 'bound_drivers': {'0': 'openvswitch'}}, 'port_security_enabled': True, 'tags': [], 'created_at': '2025-06-10T14:57:42Z', 'updated_at': '2025-06-10T14:57:47Z', 'revision_number': 5, 'project_id': '961e2027d2ac49d8b7e76065b42206ac'}] get_port /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:255

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2113923/+subscriptions