yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1385484] [NEW] Failed to destroy evacuated disks on init_host

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Fei Long Wang <flwang@xxxxxxxxxxxxxxx>
Date: Fri, 24 Oct 2014 20:48:42 -0000
Reply-to: Bug 1385484 <1385484@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

After evacuated successfully, and restarting the failed host to get it
back. User will run into below error.


<179>Sep 23 01:48:35 node-1 nova-compute 2014-09-23 01:48:35.346 13206 ERROR nova.openstack.common.threadgroup [-] error removing image
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     x.wait()
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 483, in run_service
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     service.start()
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 163, in start
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     self.manager.init_host()
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1018, in init_host
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     self._destroy_evacuated_instances(context)
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 712, in _destroy_evacuated_instances
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     bdi, destroy_disks)
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 962, in destroy
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     destroy_disks, migrate_data)
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1080, in cleanup
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     self._cleanup_rbd(instance)
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1090, in _cleanup_rbd
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     LibvirtDriver._get_rbd_driver().cleanup_volumes(instance)
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/rbd_utils.py", line 238, in cleanup_volumes
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     self.rbd.RBD().remove(client.ioctx, volume)
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/rbd.py", line 300, in remove
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     raise make_ex(ret, 'error removing image')
2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup ImageBusy: error removing image

** Affects: nova
     Importance: High
     Assignee: Fei Long Wang (flwang)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Fei Long Wang (flwang)

** Changed in: nova
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1385484

Title:
  Failed to destroy evacuated disks on init_host

Status in OpenStack Compute (Nova):
  New

Bug description:
  After evacuated successfully, and restarting the failed host to get it
  back. User will run into below error.


  <179>Sep 23 01:48:35 node-1 nova-compute 2014-09-23 01:48:35.346 13206 ERROR nova.openstack.common.threadgroup [-] error removing image
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 117, in wait
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     x.wait()
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line 49, in wait
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 168, in wait
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 194, in main
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 483, in run_service
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     service.start()
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 163, in start
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     self.manager.init_host()
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1018, in init_host
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     self._destroy_evacuated_instances(context)
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 712, in _destroy_evacuated_instances
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     bdi, destroy_disks)
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 962, in destroy
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     destroy_disks, migrate_data)
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1080, in cleanup
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     self._cleanup_rbd(instance)
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1090, in _cleanup_rbd
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     LibvirtDriver._get_rbd_driver().cleanup_volumes(instance)
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/rbd_utils.py", line 238, in cleanup_volumes
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     self.rbd.RBD().remove(client.ioctx, volume)
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/dist-packages/rbd.py", line 300, in remove
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup     raise make_ex(ret, 'error removing image')
  2014-09-23 01:48:35.346 13206 TRACE nova.openstack.common.threadgroup ImageBusy: error removing image

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1385484/+subscriptions
Follow ups

[Bug 1385484] Re: Failed to start nova-compute after evacuate
From: Alan Pevec, 2015-03-13
[Bug 1385484] Re: Failed to start nova-compute after evacuate
From: Alan Pevec, 2015-03-12
[Bug 1385484] Re: Failed to start nova-compute after evacuate
From: Thierry Carrez, 2014-12-18
[Bug 1385484] Re: Failed to start nova-compute after evacuate
From: Alan Pevec, 2014-12-05
[Bug 1385484] Re: Failed to start nova-compute after evacuate
From: Alan Pevec, 2014-12-04
[Bug 1385484] [NEW] Failed to destroy evacuated disks on init_host
From: Fei Long Wang, 2014-10-24
References

[Bug 1385484] [NEW] Failed to destroy evacuated disks on init_host
From: Fei Long Wang, 2014-10-24