← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #23690

[Bug 1380741] [NEW] Error from libvirt during destroy. Code=38 Error=Failed to terminate process with SIGKILL: Device or resource busy

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

http://logs.openstack.org/20/128020/2/check/check-tempest-dsvm-
full/6b56bf1/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-10-13_18_12_06_653

2014-10-13 18:12:06.653 ERROR nova.virt.libvirt.driver [req-017c0b4b-11ba-4f59-9895-3d0053481b00 VolumesV2ActionsTestXML-1527860598 VolumesV2ActionsTestXML-196352251] [instance: 6f733615-3124-400e-b890-b07468fa4777] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 16365 with SIGKILL: Device or resource busy
2014-10-13 18:12:08.000 ERROR nova.compute.manager [req-017c0b4b-11ba-4f59-9895-3d0053481b00 VolumesV2ActionsTestXML-1527860598 VolumesV2ActionsTestXML-196352251] [instance: 6f733615-3124-400e-b890-b07468fa4777] Setting instance vm_state to ERROR
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] Traceback (most recent call last):
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2500, in do_terminate_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     self._delete_instance(context, instance, bdms, quotas)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/hooks.py", line 131, in inner
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     rv = f(*args, **kwargs)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2469, in _delete_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     quotas.rollback()
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     six.reraise(self.type_, self.value, self.tb)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2446, in _delete_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     self._shutdown_instance(context, instance, bdms)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2376, in _shutdown_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     requested_networks)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     six.reraise(self.type_, self.value, self.tb)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2365, in _shutdown_instance
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     block_device_info)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1054, in destroy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     self._destroy(instance)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1010, in _destroy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     instance=instance)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     six.reraise(self.type_, self.value, self.tb)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 979, in _destroy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     virt_dom.destroy()
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     rv = execute(f, *args, **kwargs)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     six.reraise(c, e, tb)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     rv = meth(*args, **kwargs)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 918, in destroy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self)
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] libvirtError: Failed to terminate process 16365 with SIGKILL: Device or resource busy
2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] 


Logstash:

message:"_destroy" AND message:"libvirtError: Failed to terminate
process" AND message:"with SIGKILL: Device or resource busy" AND
module:"nova.compute.manager" AND tags:"screen-n-cpu.txt"

17 hits in 10 days, check and gate, all failures:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiX2Rlc3Ryb3lcIiBBTkQgbWVzc2FnZTpcImxpYnZpcnRFcnJvcjogRmFpbGVkIHRvIHRlcm1pbmF0ZSBwcm9jZXNzXCIgQU5EIG1lc3NhZ2U6XCJ3aXRoIFNJR0tJTEw6IERldmljZSBvciByZXNvdXJjZSBidXN5XCIgQU5EIG1vZHVsZTpcIm5vdmEuY29tcHV0ZS5tYW5hZ2VyXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1jcHUudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImN1c3RvbSIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxNC0xMC0wMVQxODo0NTowMiswMDowMCIsInRvIjoiMjAxNC0xMC0xM1QxODo0NTowMiswMDowMCIsInVzZXJfaW50ZXJ2YWwiOiIwIn0sInN0YW1wIjoxNDEzMjI2MDA1NzIxfQ==

This could be related to lvm hang bug 1373513 since it looks like the
problem is the backing volume is busy when we try to delete/destroy the
libvirt domain.

** Affects: nova
     Importance: High
         Status: Confirmed


** Tags: gate-failure libvirt volumes

** Changed in: nova
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1380741

Title:
  Error from libvirt during destroy. Code=38 Error=Failed to terminate
  process with SIGKILL: Device or resource busy

Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  http://logs.openstack.org/20/128020/2/check/check-tempest-dsvm-
  full/6b56bf1/logs/screen-n-cpu.txt.gz?level=TRACE#_2014-10-13_18_12_06_653

  2014-10-13 18:12:06.653 ERROR nova.virt.libvirt.driver [req-017c0b4b-11ba-4f59-9895-3d0053481b00 VolumesV2ActionsTestXML-1527860598 VolumesV2ActionsTestXML-196352251] [instance: 6f733615-3124-400e-b890-b07468fa4777] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 16365 with SIGKILL: Device or resource busy
  2014-10-13 18:12:08.000 ERROR nova.compute.manager [req-017c0b4b-11ba-4f59-9895-3d0053481b00 VolumesV2ActionsTestXML-1527860598 VolumesV2ActionsTestXML-196352251] [instance: 6f733615-3124-400e-b890-b07468fa4777] Setting instance vm_state to ERROR
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] Traceback (most recent call last):
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2500, in do_terminate_instance
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     self._delete_instance(context, instance, bdms, quotas)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/hooks.py", line 131, in inner
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     rv = f(*args, **kwargs)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2469, in _delete_instance
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     quotas.rollback()
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     six.reraise(self.type_, self.value, self.tb)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2446, in _delete_instance
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     self._shutdown_instance(context, instance, bdms)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2376, in _shutdown_instance
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     requested_networks)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     six.reraise(self.type_, self.value, self.tb)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2365, in _shutdown_instance
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     block_device_info)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1054, in destroy
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     self._destroy(instance)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1010, in _destroy
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     instance=instance)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in __exit__
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     six.reraise(self.type_, self.value, self.tb)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 979, in _destroy
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     virt_dom.destroy()
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     rv = execute(f, *args, **kwargs)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     six.reraise(c, e, tb)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     rv = meth(*args, **kwargs)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 918, in destroy
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777]     if ret == -1: raise libvirtError ('virDomainDestroy() failed', dom=self)
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] libvirtError: Failed to terminate process 16365 with SIGKILL: Device or resource busy
  2014-10-13 18:12:08.000 17350 TRACE nova.compute.manager [instance: 6f733615-3124-400e-b890-b07468fa4777] 

  
  Logstash:

  message:"_destroy" AND message:"libvirtError: Failed to terminate
  process" AND message:"with SIGKILL: Device or resource busy" AND
  module:"nova.compute.manager" AND tags:"screen-n-cpu.txt"

  17 hits in 10 days, check and gate, all failures:

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiX2Rlc3Ryb3lcIiBBTkQgbWVzc2FnZTpcImxpYnZpcnRFcnJvcjogRmFpbGVkIHRvIHRlcm1pbmF0ZSBwcm9jZXNzXCIgQU5EIG1lc3NhZ2U6XCJ3aXRoIFNJR0tJTEw6IERldmljZSBvciByZXNvdXJjZSBidXN5XCIgQU5EIG1vZHVsZTpcIm5vdmEuY29tcHV0ZS5tYW5hZ2VyXCIgQU5EIHRhZ3M6XCJzY3JlZW4tbi1jcHUudHh0XCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6ImN1c3RvbSIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJmcm9tIjoiMjAxNC0xMC0wMVQxODo0NTowMiswMDowMCIsInRvIjoiMjAxNC0xMC0xM1QxODo0NTowMiswMDowMCIsInVzZXJfaW50ZXJ2YWwiOiIwIn0sInN0YW1wIjoxNDEzMjI2MDA1NzIxfQ==

  This could be related to lvm hang bug 1373513 since it looks like the
  problem is the backing volume is busy when we try to delete/destroy
  the libvirt domain.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1380741/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next