yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #94785
[Bug 2085432] [NEW] Ironic builds stuck after "The mac address is in use conflict"
Public bug reported:
Release: 2023.1
Steps to reproduce:
- Precreate ports for ironic node
- Attempt to launch multiple instances at once
- You need some failure that prevents the nodes for provisioning (in my case the firewall broke PXE for one controller)
- Observe instances getting stuck in BUILD state indefinitely.
Some relevant logs:
[instance: 88d96c7a-aca6-49e4-b083-a67ed687275f] Failure
prepping block device:
ironicclient.common.apiclient.exceptions.InternalServerError: Unable to
attach VIF 5c60151c-f9ee-42cb-b2a8-f94a6a2336c8 because Ironic can not
update Neutron port 5c60151c-f9ee-42cb-b2a8-f94a6a2336c8 MAC address to
match physical MAC address 0c:42:a1:02:42:8b (HTTP 500)
Failed to update MAC address on Neutron port
5c60151c-f9ee-42cb-b2a8-f94a6a2336c8.:
openstack.exceptions.ConflictException: ConflictException: 409: Client
Error for url:
https://172.25.4.254:9696/v2.0/ports/5c60151c-f9ee-42cb-b2a8-f94a6a2336c8,
Unable to complete operation for network 6c96c4e1-7495-41ee-
ba80-a12386819bc1. The mac address 0c:42:a1:02:42:8b is in use.
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron Traceback (most
recent call last): 2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/ironic/common/neutron.py", line 176, in update_port_address
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
update_neutron_port(context, port_id, port_attrs) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/ironic/common/neutron.py", line 109, in update_neutron_port
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
client.update_port(port_id, **attrs) 2024-10-16 17:44:44.716 7 ERROR
ironic.common.neutron File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/network/v2/_proxy.py", line 2324, in update_port
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
self._update(_port.Port, port, if_revision=if_revision, 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/proxy.py",
line 64, in check 2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
return method(self, expected, actual, *args, **kwargs) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/network/v2/_proxy.py", line 150, in _update
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
res.commit(self, base_path=base_path, if_revision=if_revision)
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1742, in commit 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron return self._commit(
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1787, in _commit 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron
self._translate_response(response, has_body=has_body) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1253, in _translate_response
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
exceptions.raise_from_response(response, error_message=error_message)
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/exceptions.py", line 240, in raise_from_response
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron raise cls(
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
openstack.exceptions.ConflictException: ConflictException: 409: Client
Error for url:
https://172.25.4.254:9696/v2.0/ports/5c60151c-f9ee-42cb-b2a8-f94a6a2336c8,
Unable to complete operation for network 6c96c4e1-7495-41ee-
ba80-a12386819bc1. The mac address 0c:42:a1:02:42:8b is in use.
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
Then the during teardown:
(openstack-venv) [stackhpc_admin@SVN3-DR08-U34 openstack-config]$ openstack server event show 8c6baec9-fb74-4f68-aaa7-a203d8198251 req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | create |
| events | details='InvalidRequest', event='compute__do_build_and_run_instance', finish_time='2024-10-16T17:34:25.000000', host='svn1-dr07-u2-ironic', |
| | host_id='c04cb592b2146dc1ff0c6f0ca708218365527c67885df43ebec22320', result='Error', start_time='2024-10-16T17:34:21.000000', traceback=' File |
| | "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/utils.py", line 1439, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py", line 214, in decorated_function |
| | compute_utils.add_instance_fault_from_exc(context, |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ |
| | self.force_reraise() |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise |
| | raise self.value |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py", line 203, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py", line 2456, in _do_build_and_run_instance |
| | self._cleanup_allocated_networks(context, instance, |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py", line 2996, in _cleanup_allocated_networks |
| | self.driver.unplug_vifs(instance, network_info) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/virt/ironic/driver.py", line 1629, in unplug_vifs |
| | node = self._get_node(instance.node) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/virt/ironic/driver.py", line 221, in _get_node |
| | node = self.ironic_connection.get_node(node_id, fields=_NODE_FIELDS) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/baremetal/v1/_proxy.py", line 328, in get_node |
| | return self._get_with_fields(_node.Node, node, fields=fields) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/baremetal/v1/_proxy.py", line 63, in _get_with_fields |
| | return res.fetch( |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line 1633, in fetch |
| | request = self._prepare_request( |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line 1229, in _prepare_request |
| | raise exceptions.InvalidRequest( |
| | ' |
| id | req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90 |
| message | Error |
| project_id | 931d1604d56c45698f602ffcee907e3c |
| request_id | req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90 |
| start_time | 2024-10-16T17:34:20.000000 |
| user_id | 8072931a57a94c0f96e45a544eb0393c |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Sorry for the bad formatting.
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2085432
Title:
Ironic builds stuck after "The mac address is in use conflict"
Status in OpenStack Compute (nova):
New
Bug description:
Release: 2023.1
Steps to reproduce:
- Precreate ports for ironic node
- Attempt to launch multiple instances at once
- You need some failure that prevents the nodes for provisioning (in my case the firewall broke PXE for one controller)
- Observe instances getting stuck in BUILD state indefinitely.
Some relevant logs:
[instance: 88d96c7a-aca6-49e4-b083-a67ed687275f] Failure
prepping block device:
ironicclient.common.apiclient.exceptions.InternalServerError: Unable
to attach VIF 5c60151c-f9ee-42cb-b2a8-f94a6a2336c8 because Ironic can
not update Neutron port 5c60151c-f9ee-42cb-b2a8-f94a6a2336c8 MAC
address to match physical MAC address 0c:42:a1:02:42:8b (HTTP 500)
Failed to update MAC address on Neutron port
5c60151c-f9ee-42cb-b2a8-f94a6a2336c8.:
openstack.exceptions.ConflictException: ConflictException: 409: Client
Error for url:
https://172.25.4.254:9696/v2.0/ports/5c60151c-f9ee-42cb-b2a8-f94a6a2336c8,
Unable to complete operation for network 6c96c4e1-7495-41ee-
ba80-a12386819bc1. The mac address 0c:42:a1:02:42:8b is in use.
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron Traceback (most
recent call last): 2024-10-16 17:44:44.716 7 ERROR
ironic.common.neutron File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/ironic/common/neutron.py", line 176, in update_port_address
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
update_neutron_port(context, port_id, port_attrs) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/ironic/common/neutron.py", line 109, in update_neutron_port
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
client.update_port(port_id, **attrs) 2024-10-16 17:44:44.716 7 ERROR
ironic.common.neutron File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/network/v2/_proxy.py", line 2324, in update_port
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
self._update(_port.Port, port, if_revision=if_revision, 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/proxy.py", line 64, in check 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron return method(self,
expected, actual, *args, **kwargs) 2024-10-16 17:44:44.716 7 ERROR
ironic.common.neutron File "/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/network/v2/_proxy.py", line 150, in _update
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron return
res.commit(self, base_path=base_path, if_revision=if_revision)
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1742, in commit 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron return self._commit(
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1787, in _commit 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron
self._translate_response(response, has_body=has_body) 2024-10-16
17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/resource.py", line 1253, in _translate_response
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
exceptions.raise_from_response(response, error_message=error_message)
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron File
"/var/lib/kolla/venv/lib64/python3.9/site-
packages/openstack/exceptions.py", line 240, in raise_from_response
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron raise cls(
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
openstack.exceptions.ConflictException: ConflictException: 409: Client
Error for url:
https://172.25.4.254:9696/v2.0/ports/5c60151c-f9ee-42cb-b2a8-f94a6a2336c8,
Unable to complete operation for network 6c96c4e1-7495-41ee-
ba80-a12386819bc1. The mac address 0c:42:a1:02:42:8b is in use.
2024-10-16 17:44:44.716 7 ERROR ironic.common.neutron
Then the during teardown:
(openstack-venv) [stackhpc_admin@SVN3-DR08-U34 openstack-config]$ openstack server event show 8c6baec9-fb74-4f68-aaa7-a203d8198251 req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| action | create |
| events | details='InvalidRequest', event='compute__do_build_and_run_instance', finish_time='2024-10-16T17:34:25.000000', host='svn1-dr07-u2-ironic', |
| | host_id='c04cb592b2146dc1ff0c6f0ca708218365527c67885df43ebec22320', result='Error', start_time='2024-10-16T17:34:21.000000', traceback=' File |
| | "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/utils.py", line 1439, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py", line 214, in decorated_function |
| | compute_utils.add_instance_fault_from_exc(context, |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__ |
| | self.force_reraise() |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise |
| | raise self.value |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py", line 203, in decorated_function |
| | return function(self, context, *args, **kwargs) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py", line 2456, in _do_build_and_run_instance |
| | self._cleanup_allocated_networks(context, instance, |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/compute/manager.py", line 2996, in _cleanup_allocated_networks |
| | self.driver.unplug_vifs(instance, network_info) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/virt/ironic/driver.py", line 1629, in unplug_vifs |
| | node = self._get_node(instance.node) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/nova/virt/ironic/driver.py", line 221, in _get_node |
| | node = self.ironic_connection.get_node(node_id, fields=_NODE_FIELDS) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/baremetal/v1/_proxy.py", line 328, in get_node |
| | return self._get_with_fields(_node.Node, node, fields=fields) |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/baremetal/v1/_proxy.py", line 63, in _get_with_fields |
| | return res.fetch( |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line 1633, in fetch |
| | request = self._prepare_request( |
| | File "/var/lib/kolla/venv/lib64/python3.9/site-packages/openstack/resource.py", line 1229, in _prepare_request |
| | raise exceptions.InvalidRequest( |
| | ' |
| id | req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90 |
| message | Error |
| project_id | 931d1604d56c45698f602ffcee907e3c |
| request_id | req-f2daa91e-0011-4e2f-87e4-5e7e61a17f90 |
| start_time | 2024-10-16T17:34:20.000000 |
| user_id | 8072931a57a94c0f96e45a544eb0393c |
+------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Sorry for the bad formatting.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2085432/+subscriptions
