yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2097586] [NEW] Random test failures with libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2097586@xxxxxxxxxxxxxxxxxx>
Date: Fri, 07 Feb 2025 06:10:45 -0000
Reply-to: Bug 2097586 <2097586@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Test failure during vm destroy as:-
2025-01-31 11:57:58,135 120175 INFO     [tempest.common.waiters] State transition "RESIZE/resize_prep" ==> "RESIZE/resize_migrating" after 8 second wait

2025-01-31 11:58:01,827 120175 INFO     [tempest.common.waiters] State
transition "RESIZE/resize_migrating" ==> "RESIZE/resize_migrated" after
11 second wait

2025-01-31 11:58:05,489 120175 INFO     [tempest.common.waiters] State
transition "RESIZE/resize_migrated" ==> "RESIZE/resize_finish" after 15
second wait

2025-01-31 11:58:15,869 120175 INFO     [tempest.common.waiters] State
transition "RESIZE/resize_finish" ==> "VERIFY_RESIZE/None" after 25
second wait


    Response - Headers: {'date': 'Fri, 31 Jan 2025 11:59:10 GMT', 'server': 'Apache/2.4.58 (Ubuntu)', 'content-length': '1718', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.72', 'x-openstack-nova-api-version': '2.72', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-d15679f5-e820-42ac-b93e-76024fa41940', 'x-compute-request-id': 'req-d15679f5-e820-42ac-b93e-76024fa41940', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed'}
        Body: b'{"server": {"id": "007a11e4-32d4-4b1c-9199-56a664fe86ed", "name": "tempest-QoSBandwidthAndPacketRateTests-server-1599599327", "status": "ERROR", "tenant_id": "fab8f5903cec400ab103cf64a8530b57", "user_id": "35b1e65e2c354a48b77f2f7247070fde", "metadata": {}, "hostId": "8ac31065c7b89285e06e31bd99c60023f629010753c70c6f324d93ea", "image": {"id": "acb24912-db2b-4ee1-8a71-9ce02d0abf0f", "links": [{"rel": "bookmark", "href": "https://158.69.67.245/compute/images/acb24912-db2b-4ee1-8a71-9ce02d0abf0f"}]}, "flavor": {"vcpus": 1, "ram": 192, "disk": 2, "ephemeral": 0, "swap": 0, "original_name": "m1.nanoextra-391323443", "extra_specs": {}}, "created": "2025-01-31T11:57:38Z", "updated": "2025-01-31T11:59:09Z", "addresses": {"tempest-network-smoke--673944778": [{"version": 4, "addr": "10.1.0.22", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:60:17:c9"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}, {"rel": "bookmark", "href": "https://158.69.67.245/compute/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2025-01-31T11:59:09Z", "message": "libvirtError"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2025-01-31T11:58:13.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "default"}], "OS-EXT-STS:task_state": "resize_reverting", "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": null, "server_groups": []}}'
2025-01-31 11:59:10,617 120175 INFO     [tempest.common.waiters] State transition "REVERT_RESIZE/resize_reverting" ==> "ERROR/resize_reverting" after 54 second wait


Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/common/waiters.py", line 136, in wait_for_server_termination
    raise lib_exc.DeleteErrorException(details, server_id=server_id)
tempest.lib.exceptions.DeleteErrorException: Resource %(resource_id)s failed to delete and is in ERROR status
Details: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to delete and is in ERROR status. Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message': 'libvirtError'}.
}}}

Traceback (most recent call last):
  File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 65, in wrapper
    return f(*func_args, **func_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/stack/tempest/tempest/scenario/test_network_qos_placement.py", line 851, in test_server_resize_revert
    waiters.wait_for_server_status(
  File "/opt/stack/tempest/tempest/common/waiters.py", line 80, in wait_for_server_status
    raise exceptions.BuildErrorException(details, server_id=server_id)
tempest.exceptions.BuildErrorException: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to build and is in ERROR status
Details: Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message': 'libvirtError'}.


Nova compute logs:-
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: WARNING nova.virt.libvirt.driver [None req-5e44a4a9-7f73-4019-aec0-86bec9372149 tempest-QoSBandwidthAndPacketRateTests-1787381104 tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Cannot destroy instance, general system call failure: libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [None req-5e44a4a9-7f73-4019-aec0-86bec9372149 tempest-QoSBandwidthAndPacketRateTests-1787381104 tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Setting instance vm_state to ERROR: libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Traceback (most recent call last):
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/compute/manager.py", line 11382, in _error_out_instance_on_exception
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     yield
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/compute/manager.py", line 5914, in revert_resize
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     self.driver.destroy(context, instance, network_info,
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1622, in destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     self._destroy(instance)
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1567, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     with excutils.save_and_reraise_exception():
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     self.force_reraise()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     raise self.value
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1535, in _destroy
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     guest.poweroff()
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 173, in poweroff
Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     self._domain.destroy()
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 186, in doit
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     result = proxy_call(self._autowrap, f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 144, in proxy_call
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     rv = execute(f, *args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 125, in execute
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     raise e.with_traceback(tb)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 82, in tworker
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     rv = meth(*args, **kwargs)
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]          ^^^^^^^^^^^^^^^^^^^^^
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/usr/lib/python3/dist-packages/libvirt.py", line 1461, in destroy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     raise libvirtError('virDomainDestroy() failed')
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]

Seen twice, ex builds:-
https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html
https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html

Opensearch(creds: openstack/openstack)
https://opensearch.logs.openstack.org/_dashboards/app/data-
explorer/discover?security_tenant=global#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_q=(filters:!(),query:(language:kuery,query:'message:%22libvirt.libvirtError:%20Failed%20to%20terminate%20process%22'))


Found an old bug https://bugs.launchpad.net/nova/+bug/1353939 under which a workaround was applied to retry 6 times before giving up but that was targetting libvirt < 4.7.0 with https://review.opendev.org/c/openstack/nova/+/639091. And then it was removed with support removal of old libvirt versions https://review.opendev.org/c/openstack/nova/+/746984.

This bug is to track what exactly triggering this behavior now and if
some workaround flag can be added in nova to have some attempts on
temporary failure during destroy as before.

Also some discussion on nova channel
https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-
nova.2025-02-06.log.html#t2025-02-06T15:04:13

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2097586

Title:
  Random test failures with libvirt.libvirtError: Failed to terminate
  process 53286 with SIGKILL: Device or resource busy

Status in OpenStack Compute (nova):
  New

Bug description:
  Test failure during vm destroy as:-
  2025-01-31 11:57:58,135 120175 INFO     [tempest.common.waiters] State transition "RESIZE/resize_prep" ==> "RESIZE/resize_migrating" after 8 second wait

  2025-01-31 11:58:01,827 120175 INFO     [tempest.common.waiters] State
  transition "RESIZE/resize_migrating" ==> "RESIZE/resize_migrated"
  after 11 second wait

  2025-01-31 11:58:05,489 120175 INFO     [tempest.common.waiters] State
  transition "RESIZE/resize_migrated" ==> "RESIZE/resize_finish" after
  15 second wait

  2025-01-31 11:58:15,869 120175 INFO     [tempest.common.waiters] State
  transition "RESIZE/resize_finish" ==> "VERIFY_RESIZE/None" after 25
  second wait

  
      Response - Headers: {'date': 'Fri, 31 Jan 2025 11:59:10 GMT', 'server': 'Apache/2.4.58 (Ubuntu)', 'content-length': '1718', 'content-type': 'application/json', 'openstack-api-version': 'compute 2.72', 'x-openstack-nova-api-version': '2.72', 'vary': 'OpenStack-API-Version,X-OpenStack-Nova-API-Version', 'x-openstack-request-id': 'req-d15679f5-e820-42ac-b93e-76024fa41940', 'x-compute-request-id': 'req-d15679f5-e820-42ac-b93e-76024fa41940', 'connection': 'close', 'status': '200', 'content-location': 'https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed'}
          Body: b'{"server": {"id": "007a11e4-32d4-4b1c-9199-56a664fe86ed", "name": "tempest-QoSBandwidthAndPacketRateTests-server-1599599327", "status": "ERROR", "tenant_id": "fab8f5903cec400ab103cf64a8530b57", "user_id": "35b1e65e2c354a48b77f2f7247070fde", "metadata": {}, "hostId": "8ac31065c7b89285e06e31bd99c60023f629010753c70c6f324d93ea", "image": {"id": "acb24912-db2b-4ee1-8a71-9ce02d0abf0f", "links": [{"rel": "bookmark", "href": "https://158.69.67.245/compute/images/acb24912-db2b-4ee1-8a71-9ce02d0abf0f"}]}, "flavor": {"vcpus": 1, "ram": 192, "disk": 2, "ephemeral": 0, "swap": 0, "original_name": "m1.nanoextra-391323443", "extra_specs": {}}, "created": "2025-01-31T11:57:38Z", "updated": "2025-01-31T11:59:09Z", "addresses": {"tempest-network-smoke--673944778": [{"version": 4, "addr": "10.1.0.22", "OS-EXT-IPS:type": "fixed", "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:60:17:c9"}]}, "accessIPv4": "", "accessIPv6": "", "links": [{"rel": "self", "href": "https://158.69.67.245/compute/v2.1/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}, {"rel": "bookmark", "href": "https://158.69.67.245/compute/servers/007a11e4-32d4-4b1c-9199-56a664fe86ed"}], "OS-DCF:diskConfig": "MANUAL", "fault": {"code": 500, "created": "2025-01-31T11:59:09Z", "message": "libvirtError"}, "OS-EXT-AZ:availability_zone": "nova", "config_drive": "", "key_name": null, "OS-SRV-USG:launched_at": "2025-01-31T11:58:13.000000", "OS-SRV-USG:terminated_at": null, "security_groups": [{"name": "default"}], "OS-EXT-STS:task_state": "resize_reverting", "OS-EXT-STS:vm_state": "error", "OS-EXT-STS:power_state": 1, "os-extended-volumes:volumes_attached": [], "locked": false, "description": null, "tags": [], "trusted_image_certificates": null, "server_groups": []}}'
  2025-01-31 11:59:10,617 120175 INFO     [tempest.common.waiters] State transition "REVERT_RESIZE/resize_reverting" ==> "ERROR/resize_reverting" after 54 second wait

  
  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/common/waiters.py", line 136, in wait_for_server_termination
      raise lib_exc.DeleteErrorException(details, server_id=server_id)
  tempest.lib.exceptions.DeleteErrorException: Resource %(resource_id)s failed to delete and is in ERROR status
  Details: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to delete and is in ERROR status. Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message': 'libvirtError'}.
  }}}

  Traceback (most recent call last):
    File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 65, in wrapper
      return f(*func_args, **func_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/opt/stack/tempest/tempest/scenario/test_network_qos_placement.py", line 851, in test_server_resize_revert
      waiters.wait_for_server_status(
    File "/opt/stack/tempest/tempest/common/waiters.py", line 80, in wait_for_server_status
      raise exceptions.BuildErrorException(details, server_id=server_id)
  tempest.exceptions.BuildErrorException: Server 007a11e4-32d4-4b1c-9199-56a664fe86ed failed to build and is in ERROR status
  Details: Fault: {'code': 500, 'created': '2025-01-31T11:59:09Z', 'message': 'libvirtError'}.

  
  Nova compute logs:-
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: WARNING nova.virt.libvirt.driver [None req-5e44a4a9-7f73-4019-aec0-86bec9372149 tempest-QoSBandwidthAndPacketRateTests-1787381104 tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Cannot destroy instance, general system call failure: libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [None req-5e44a4a9-7f73-4019-aec0-86bec9372149 tempest-QoSBandwidthAndPacketRateTests-1787381104 tempest-QoSBandwidthAndPacketRateTests-1787381104-project-member] [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Setting instance vm_state to ERROR: libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] Traceback (most recent call last):
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/compute/manager.py", line 11382, in _error_out_instance_on_exception
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     yield
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/compute/manager.py", line 5914, in revert_resize
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     self.driver.destroy(context, instance, network_info,
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1622, in destroy
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     self._destroy(instance)
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1567, in _destroy
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     with excutils.save_and_reraise_exception():
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     self.force_reraise()
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     raise self.value
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1535, in _destroy
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     guest.poweroff()
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 173, in poweroff
  Jan 31 11:59:04.005181 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     self._domain.destroy()
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 186, in doit
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     rv = execute(f, *args, **kwargs)
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 125, in execute
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     raise e.with_traceback(tb)
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/opt/stack/data/venv/lib/python3.12/site-packages/eventlet/tpool.py", line 82, in tworker
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     rv = meth(*args, **kwargs)
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]          ^^^^^^^^^^^^^^^^^^^^^
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]   File "/usr/lib/python3/dist-packages/libvirt.py", line 1461, in destroy
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]     raise libvirtError('virDomainDestroy() failed')
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed] libvirt.libvirtError: Failed to terminate process 53286 with SIGKILL: Device or resource busy
  Jan 31 11:59:04.212284 np0039731693 nova-compute[31800]: ERROR nova.compute.manager [instance: 007a11e4-32d4-4b1c-9199-56a664fe86ed]

  Seen twice, ex builds:-
  https://0c63ab9652170854bf26-a09d1a3317eb4b9b558e42ad19c25861.ssl.cf2.rackcdn.com/940474/1/gate/neutron-ovs-tempest-multinode-full/5184c7f/testr_results.html
  https://97ad4d1320a89f3380ef-01f5fc3a5734547a13a0f54725d40b32.ssl.cf5.rackcdn.com/936364/4/gate/nova-multi-cell/3003ee6/testr_results.html

  Opensearch(creds: openstack/openstack)
  https://opensearch.logs.openstack.org/_dashboards/app/data-
  explorer/discover?security_tenant=global#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'94869730-aea8-11ec-9e6a-83741af3fdcd',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-30d,to:now))&_q=(filters:!(),query:(language:kuery,query:'message:%22libvirt.libvirtError:%20Failed%20to%20terminate%20process%22'))

  
  Found an old bug https://bugs.launchpad.net/nova/+bug/1353939 under which a workaround was applied to retry 6 times before giving up but that was targetting libvirt < 4.7.0 with https://review.opendev.org/c/openstack/nova/+/639091. And then it was removed with support removal of old libvirt versions https://review.opendev.org/c/openstack/nova/+/746984.

  This bug is to track what exactly triggering this behavior now and if
  some workaround flag can be added in nova to have some attempts on
  temporary failure during destroy as before.

  Also some discussion on nova channel
  https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-
  nova.2025-02-06.log.html#t2025-02-06T15:04:13

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2097586/+subscriptions