yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95990
[Bug 2113923] [NEW] Neutron metadata agent fails to register in Epoxy
Public bug reported:
Neutron Metadata agent fails to register in the Epoxy release.
On a fresh deployment, it just never registers
When upgrading from Caracal, it gets marked as down because no heartbeats are sent after the upgrade.
I have not tested Dalmatian.
Even though heartbeats fail, it still works fine. It responds and
requests can be seen in the logs.
Tested in a Kayobe AIO (Kolla containers)
Tested on Rocky 9 and Ubuntu Noble, both are affected
OVN is unaffected
Appears to have affected Kolla-Ansible since mid-march:
https://review.opendev.org/c/openstack/kolla-ansible/+/944194
Severity is high. No functional issues but lots of alerts firing .
Logs
====
Neutron server error (last heartbeat is time of upgrade):
Agent healthcheck: found 1 dead agents out of 4:
Type Last heartbeat host
Metadata agent 2025-06-09 13:29:05 alex-aio-epoxy-rocky06.novalocal
Openstack view after upgrade:
$ openstack network agent list -c "Agent Type" -c Alive -c State -c Binary
+--------------------+-------+-------+---------------------------+
| Agent Type | Alive | State | Binary |
+--------------------+-------+-------+---------------------------+
| Open vSwitch agent | :-) | UP | neutron-openvswitch-agent |
| DHCP agent | :-) | UP | neutron-dhcp-agent |
| L3 agent | :-) | UP | neutron-l3-agent |
| Metadata agent | XXX | UP | neutron-metadata-agent |
+--------------------+-------+-------+---------------------------+
Neutron metadata agent response log:
2025-06-10 15:18:24.306 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Request: GET /latest/meta-data/public-ipv4/ HTTP/1.1
host: 169.254.169.254
user-agent: curl/7.84.0
accept: */*
x-neutron-network-id: 03739e2a-f552-43b1-9d6f-43db61d03aaf
x-forwarded-for: 10.0.0.59
connection: close
handle /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:163
2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Acquiring lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:405
2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" acquired by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: waited 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:410
2025-06-10 15:18:24.311 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" "released" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: held 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:424
2025-06-10 15:18:24.833 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Got ports for remote_address 10.0.0.59, network_id 03739e2a-f552-43b1-9d6f-43db61d03aaf, remote_mac None, router_id None[{'id': 'cdbb6292-e301-4f5c-bc34-33e139eb97fb', 'name': '', 'network_id': '03739e2a-f552-43b1-9d6f-43db61d03aaf', 'tenant_id': '961e2027d2ac49d8b7e76065b42206ac', 'mac_address': 'fa:16:3e:d5:a3:a6', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'b68cc942-cfc2-4ee7-be20-5ce5e1f62353', 'device_owner': 'compute:nova', 'standard_attr_id': 662, 'fixed_ips': [{'subnet_id': 'c0cfe0fc-f3af-4ac1-9e11-a4fbbcd31167', 'ip_address': '10.0.0.59'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['f2f888b6-1209-4127-8981-acec08ec4eed'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'alex-aio-epoxy-rocky-7.novalocal', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system', 'bridge_name': 'br-int', 'bound_drivers': {'0': 'openvswitch'}}, 'port_security_enabled': True, 'tags': [], 'created_at': '2025-06-10T14:57:42Z', 'updated_at': '2025-06-10T14:57:47Z', 'revision_number': 5, 'project_id': '961e2027d2ac49d8b7e76065b42206ac'}] get_port /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:255
** Affects: neutron
Importance: Undecided
Status: New
** Tags: ovs
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2113923
Title:
Neutron metadata agent fails to register in Epoxy
Status in neutron:
New
Bug description:
Neutron Metadata agent fails to register in the Epoxy release.
On a fresh deployment, it just never registers
When upgrading from Caracal, it gets marked as down because no heartbeats are sent after the upgrade.
I have not tested Dalmatian.
Even though heartbeats fail, it still works fine. It responds and
requests can be seen in the logs.
Tested in a Kayobe AIO (Kolla containers)
Tested on Rocky 9 and Ubuntu Noble, both are affected
OVN is unaffected
Appears to have affected Kolla-Ansible since mid-march:
https://review.opendev.org/c/openstack/kolla-ansible/+/944194
Severity is high. No functional issues but lots of alerts firing .
Logs
====
Neutron server error (last heartbeat is time of upgrade):
Agent healthcheck: found 1 dead agents out of 4:
Type Last heartbeat host
Metadata agent 2025-06-09 13:29:05 alex-aio-epoxy-rocky06.novalocal
Openstack view after upgrade:
$ openstack network agent list -c "Agent Type" -c Alive -c State -c Binary
+--------------------+-------+-------+---------------------------+
| Agent Type | Alive | State | Binary |
+--------------------+-------+-------+---------------------------+
| Open vSwitch agent | :-) | UP | neutron-openvswitch-agent |
| DHCP agent | :-) | UP | neutron-dhcp-agent |
| L3 agent | :-) | UP | neutron-l3-agent |
| Metadata agent | XXX | UP | neutron-metadata-agent |
+--------------------+-------+-------+---------------------------+
Neutron metadata agent response log:
2025-06-10 15:18:24.306 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Request: GET /latest/meta-data/public-ipv4/ HTTP/1.1
host: 169.254.169.254
user-agent: curl/7.84.0
accept: */*
x-neutron-network-id: 03739e2a-f552-43b1-9d6f-43db61d03aaf
x-forwarded-for: 10.0.0.59
connection: close
handle /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:163
2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Acquiring lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:405
2025-06-10 15:18:24.309 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" acquired by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: waited 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:410
2025-06-10 15:18:24.311 7 DEBUG oslo_concurrency.lockutils [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Lock "oslo_read_shm_alex-aio-epoxy-rocky-7.novalocal_neutron-metadata-agent" "released" by "oslo_messaging._drivers.amqpdriver.QManager.get.<locals>.read_from_shm" :: held 0.001s inner /var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_concurrency/lockutils.py:424
2025-06-10 15:18:24.833 7 DEBUG neutron.agent.metadata.agent [None req-49747846-774c-40ad-9fef-0eaaf7ea1705 - - - - - -] Got ports for remote_address 10.0.0.59, network_id 03739e2a-f552-43b1-9d6f-43db61d03aaf, remote_mac None, router_id None[{'id': 'cdbb6292-e301-4f5c-bc34-33e139eb97fb', 'name': '', 'network_id': '03739e2a-f552-43b1-9d6f-43db61d03aaf', 'tenant_id': '961e2027d2ac49d8b7e76065b42206ac', 'mac_address': 'fa:16:3e:d5:a3:a6', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'b68cc942-cfc2-4ee7-be20-5ce5e1f62353', 'device_owner': 'compute:nova', 'standard_attr_id': 662, 'fixed_ips': [{'subnet_id': 'c0cfe0fc-f3af-4ac1-9e11-a4fbbcd31167', 'ip_address': '10.0.0.59'}], 'allowed_address_pairs': [], 'extra_dhcp_opts': [], 'security_groups': ['f2f888b6-1209-4127-8981-acec08ec4eed'], 'description': '', 'binding:vnic_type': 'normal', 'binding:profile': {}, 'binding:host_id': 'alex-aio-epoxy-rocky-7.novalocal', 'binding:vif_type': 'ovs', 'binding:vif_details': {'connectivity': 'l2', 'port_filter': True, 'ovs_hybrid_plug': True, 'datapath_type': 'system', 'bridge_name': 'br-int', 'bound_drivers': {'0': 'openvswitch'}}, 'port_security_enabled': True, 'tags': [], 'created_at': '2025-06-10T14:57:42Z', 'updated_at': '2025-06-10T14:57:47Z', 'revision_number': 5, 'project_id': '961e2027d2ac49d8b7e76065b42206ac'}] get_port /var/lib/kolla/venv/lib64/python3.9/site-packages/neutron/agent/metadata/agent.py:255
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2113923/+subscriptions