yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95324
[Bug 2097586] [NEW] Random test failures with libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Public bug reported:
Test failure during vm destroy as:-
2025-01-31 11:57:58,135 120175 INFO [tempest.common.waiters] State transition "RESIZE/resize_prep" ==> "RESIZE/resize_migrating" after 8 second wait
2025-01-31 11:58:01,827 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_migrating" ==> "RESIZE/resize_migrated" after
11 second wait
2025-01-31 11:58:05,489 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_migrated" ==> "RESIZE/resize_finish" after 15
second wait
2025-01-31 11:58:15,869 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_finish" ==> "VERIFY_RESIZE/None" after 25
second wait
Response - Headers: {'date': 'Fri, 31 Jan 2025 11:59:10 GMT', 'server': 'Apache/2.4.58 (Ubuntu)', 'content-length': '1718', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.72', 'x-openstack-nova-api-version': '2.72', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-d15679f5-e820-42ac-b93e-76024fa41940', 'x-compute-request-id': 'req-d15679f5-e820-42ac-b93e-76024fa41940', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed'}
Body: b'{"server": {"id": "007a11e4-32d4-4b1c-9199-56a664fe86ed", "name": "tempest-QoSBandwidthAndPacketRateTests-server-1599599327", "status": "ERROR", "tenant_id": "fab8f5903cec400ab103cf64a8530b57", "user_id": "35b1e65e2c354a48b77f2f7247070fde", "metadata": {}, "hostId": "8ac31065c7b89285e06e31bd99c60023f629010753c70c6f324d93ea", "image": {"id": "acb24912-db2b-4ee1-8a71-9ce02d0abf0f", "links": [{"rel": "bookmark", "href": "https://158.69.67.245/compute/images/acb24912-db2b-4ee1-8a71-9ce02d0abf0f"}]}, "flavor": {"vcpus": 1, "ram": 192, "disk": 2, "ephemeral": 0, "swap": 0, "original_name": "m1.nanoextra-391323443", "extra_specs": {}}, "created": "2025-01-31T11:57:38Z", "updated": "2025-01-31T11:59:09Z", "addresses": {"tempest-network-smoke--673944778": [{"version": 4, "addr": "10.1.0.22", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:60:17:c9"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}, {"rel": "bookmark", "href": "https://158.69.67.245/compute/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2025-01-31T11:59:09Z", "message": "libvirtError"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2025-01-31T11:58:13.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "default"}], "OS-EXT-STS:task_state": "resize_reverting", "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": null, "server_groups": []}}'
2025-01-31 11:59:10,617 120175 INFO [tempest.common.waiters] State transition "REVERT_RESIZE/resize_reverting" ==> "ERROR/resize_reverting" after 54 second wait
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/waiters.py", line 136, in wait_for_server_termination
raise lib_exc.DeleteErrorException(details, server_id=server_id)
tempest.lib.exceptions.DeleteErrorException: Resource %(resource_id)s failed to delete and is in ERROR status
Details: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to delete and is in ERROR status. Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message': 'libvirtError'}.
}}}
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 65, in wrapper
return f(*func_args, **func_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/stack/tempest/tempest/scenario/test_network_qos_placement.py", line 851, in test_server_resize_revert
waiters.wait_for_server_status(
File "/opt/stack/tempest/tempest/common/waiters.py", line 80, in wait_for_server_status
raise exceptions.BuildErrorException(details, server_id=server_id)
tempest.exceptions.BuildErrorException: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to build and is in ERROR status
Details: Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message': 'libvirtError'}.
Nova compute logs:-
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: WARNING nova.virt.libvirt.driver [None req-5e44a4a9-7f73-4019-aec0-86bec9372149 tempest-QoSBandwidthAndPacketRateTests-1787381104 tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Cannot destroy instance, general system call failure: libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [None req-5e44a4a9-7f73-4019-aec0-86bec9372149 tempest-QoSBandwidthAndPacketRateTests-1787381104 tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Setting instance vm_state to ERROR: libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Traceback (most recent call last):
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/compute/manager.py", line 11382, in _error_out_instance_on_exception
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] yield
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/compute/manager.py", line 5914, in revert_resize
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] self.driver.destroy(context, instance, network_info,
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1622, in destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] self._destroy(instance)
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1567, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] with excutils.save_and_reraise_exception():
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] self.force_reraise()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise self.value
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1535, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] guest.poweroff()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 173, in poweroff
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] self._domain.destroy()
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 186, in doit
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] result = proxy_call(self._autowrap, f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 144, in proxy_call
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] rv = execute(f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 125, in execute
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise e.with_traceback(tb)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 82, in tworker
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] rv = meth(*args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] ^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/usr/lib/python3/dist-packages/libvirt.py", line 1461, in destroy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise libvirtError('virDomainDestroy() failed')
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
Seen twice, ex builds:-
https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html
https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html
Opensearch(creds: openstack/openstack)
https://opensearch.logs.openstack.org/_dashboards/app/data-
explorer/discover?security_tenant=global#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_q=(filters:!(),query:(language:kuery,query:'message:%22libvirt.libvirtError:%20Failed%20to%20terminate%20process%22'))
Found an old bug https://bugs.launchpad.net/nova/+bug/1353939 under which a workaround was applied to retry 6 times before giving up but that was targetting libvirt < 4.7.0 with https://review.opendev.org/c/openstack/nova/+/639091. And then it was removed with support removal of old libvirt versions https://review.opendev.org/c/openstack/nova/+/746984.
This bug is to track what exactly triggering this behavior now and if
some workaround flag can be added in nova to have some attempts on
temporary failure during destroy as before.
Also some discussion on nova channel
https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-
nova.2025-02-06.log.html#t2025-02-06T15:04:13
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2097586
Title:
Random test failures with libvirt.libvirtError: Failed to terminate
process 53286 with SIGKILL: Device or resource busy
Status in OpenStack Compute (nova):
New
Bug description:
Test failure during vm destroy as:-
2025-01-31 11:57:58,135 120175 INFO [tempest.common.waiters] State transition "RESIZE/resize_prep" ==> "RESIZE/resize_migrating" after 8 second wait
2025-01-31 11:58:01,827 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_migrating" ==> "RESIZE/resize_migrated"
after 11 second wait
2025-01-31 11:58:05,489 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_migrated" ==> "RESIZE/resize_finish" after
15 second wait
2025-01-31 11:58:15,869 120175 INFO [tempest.common.waiters] State
transition "RESIZE/resize_finish" ==> "VERIFY_RESIZE/None" after 25
second wait
Response - Headers: {'date': 'Fri, 31 Jan 2025 11:59:10 GMT', 'server': 'Apache/2.4.58 (Ubuntu)', 'content-length': '1718', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.72', 'x-openstack-nova-api-version': '2.72', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-d15679f5-e820-42ac-b93e-76024fa41940', 'x-compute-request-id': 'req-d15679f5-e820-42ac-b93e-76024fa41940', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed'}
Body: b'{"server": {"id": "007a11e4-32d4-4b1c-9199-56a664fe86ed", "name": "tempest-QoSBandwidthAndPacketRateTests-server-1599599327", "status": "ERROR", "tenant_id": "fab8f5903cec400ab103cf64a8530b57", "user_id": "35b1e65e2c354a48b77f2f7247070fde", "metadata": {}, "hostId": "8ac31065c7b89285e06e31bd99c60023f629010753c70c6f324d93ea", "image": {"id": "acb24912-db2b-4ee1-8a71-9ce02d0abf0f", "links": [{"rel": "bookmark", "href": "https://158.69.67.245/compute/images/acb24912-db2b-4ee1-8a71-9ce02d0abf0f"}]}, "flavor": {"vcpus": 1, "ram": 192, "disk": 2, "ephemeral": 0, "swap": 0, "original_name": "m1.nanoextra-391323443", "extra_specs": {}}, "created": "2025-01-31T11:57:38Z", "updated": "2025-01-31T11:59:09Z", "addresses": {"tempest-network-smoke--673944778": [{"version": 4, "addr": "10.1.0.22", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:60:17:c9"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}, {"rel": "bookmark", "href": "https://158.69.67.245/compute/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2025-01-31T11:59:09Z", "message": "libvirtError"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2025-01-31T11:58:13.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "default"}], "OS-EXT-STS:task_state": "resize_reverting", "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": null, "server_groups": []}}'
2025-01-31 11:59:10,617 120175 INFO [tempest.common.waiters] State transition "REVERT_RESIZE/resize_reverting" ==> "ERROR/resize_reverting" after 54 second wait
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/waiters.py", line 136, in wait_for_server_termination
raise lib_exc.DeleteErrorException(details, server_id=server_id)
tempest.lib.exceptions.DeleteErrorException: Resource %(resource_id)s failed to delete and is in ERROR status
Details: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to delete and is in ERROR status. Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message': 'libvirtError'}.
}}}
Traceback (most recent call last):
File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 65, in wrapper
return f(*func_args, **func_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/stack/tempest/tempest/scenario/test_network_qos_placement.py", line 851, in test_server_resize_revert
waiters.wait_for_server_status(
File "/opt/stack/tempest/tempest/common/waiters.py", line 80, in wait_for_server_status
raise exceptions.BuildErrorException(details, server_id=server_id)
tempest.exceptions.BuildErrorException: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to build and is in ERROR status
Details: Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message': 'libvirtError'}.
Nova compute logs:-
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: WARNING nova.virt.libvirt.driver [None req-5e44a4a9-7f73-4019-aec0-86bec9372149 tempest-QoSBandwidthAndPacketRateTests-1787381104 tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Cannot destroy instance, general system call failure: libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [None req-5e44a4a9-7f73-4019-aec0-86bec9372149 tempest-QoSBandwidthAndPacketRateTests-1787381104 tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Setting instance vm_state to ERROR: libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Traceback (most recent call last):
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/compute/manager.py", line 11382, in _error_out_instance_on_exception
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] yield
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/compute/manager.py", line 5914, in revert_resize
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] self.driver.destroy(context, instance, network_info,
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1622, in destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] self._destroy(instance)
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1567, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] with excutils.save_and_reraise_exception():
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] self.force_reraise()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise self.value
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1535, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] guest.poweroff()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 173, in poweroff
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] self._domain.destroy()
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 186, in doit
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] result = proxy_call(self._autowrap, f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 144, in proxy_call
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] rv = execute(f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 125, in execute
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise e.with_traceback(tb)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 82, in tworker
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] rv = meth(*args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] ^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] File "/usr/lib/python3/dist-packages/libvirt.py", line 1461, in destroy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] raise libvirtError('virDomainDestroy() failed')
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]
Seen twice, ex builds:-
https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html
https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html
Opensearch(creds: openstack/openstack)
https://opensearch.logs.openstack.org/_dashboards/app/data-
explorer/discover?security_tenant=global#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_q=(filters:!(),query:(language:kuery,query:'message:%22libvirt.libvirtError:%20Failed%20to%20terminate%20process%22'))
Found an old bug https://bugs.launchpad.net/nova/+bug/1353939 under which a workaround was applied to retry 6 times before giving up but that was targetting libvirt < 4.7.0 with https://review.opendev.org/c/openstack/nova/+/639091. And then it was removed with support removal of old libvirt versions https://review.opendev.org/c/openstack/nova/+/746984.
This bug is to track what exactly triggering this behavior now and if
some workaround flag can be added in nova to have some attempts on
temporary failure during destroy as before.
Also some discussion on nova channel
https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-
nova.2025-02-06.log.html#t2025-02-06T15:04:13
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2097586/+subscriptions