yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1565859] Re: Can't detach SVC volume from an instance

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Mon, 04 Apr 2016 20:14:25 -0000
Reply-to: Bug 1565859 <1565859@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Nevermind about detaching the volume from cinder, the compute manager
tries to detach in the virt driver first, and then when that is done the
compute manager terminates the connection via cinder API and then
detaches the volume via cinder API.

Following:

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4785

Do you see anything in the libvirtd logs for the volume/instance uuid
here?

** Changed in: cinder
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1565859

Title:
  Can't detach SVC volume from an instance; guest detach device times
  out

Status in Cinder:
  Invalid
Status in OpenStack Compute (nova):
  Incomplete

Bug description:
  Steps to Reproduce:
  1. Setup instance with a libvirt/KVM host and SVC storage
  2. Spawn a VM
  3. Attach a volume to the VM
  4. Wait for volume attachment to complete successfully
  5. Detach the volume

  Expected Result:
  1. The volume is detached from the VM
  2. The volume's status becomes "Available"
  3. The volume can be deleted.

  Actual result:
  1. Volume remains attached to the VM (waited over 10 minutes)
  2. The volume's state stays "In-Use"

  Logs:
  016-03-24 16:34:13.852 143842 INFO nova.compute.resource_tracker [-] Final resource view: name=C387f19U21-KVM phys_ram=260533MB used_ram=4608MB phys_disk=545GB used_disk=40GB total_vcpus=160 used_vcpus=2 pci_stats=[]
  2016-03-24 16:34:14.081 143842 INFO nova.compute.resource_tracker [-] Compute_service record updated for C387f19U21_KVM:C387f19U21-KVM
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall [-] Dynamic interval looping call 'oslo_service.loopingcall._func' failed
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall Traceback (most recent call last):
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 136, in _run_loop
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall     result = func(*self.args, **self.kw)
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 377, in _func
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall     return self._sleep_time
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall     self.force_reraise()
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall     six.reraise(self.type_, self.value, self.tb)
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 356, in _func
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall     result = f(*args, **kwargs)
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 342, in _do_wait_and_retry_detach
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall     reason=reason)
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain.)
  2016-03-24 16:34:44.919 143842 ERROR oslo.service.loopingcall
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [req-fab1608e-cffe-40f9-82d0-a4c7a9cebf10 3ebcf1a38bc7b4977b7f8da32faad97bdef843372a670bb2817f8a066f042b9b e10bc17f58d8499a8fab1b05687123e5 - - -] [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] Failed to detach volume 4221ccad-0f98-4f78-ad06-92ea4941afc1 from /dev/vdb
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] Traceback (most recent call last):
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4767, in _driver_detach_volume
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     encryption=encryption)
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1469, in detach_volume
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     wait_for_detach()
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 385, in func
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     return evt.wait()
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     return hubs.get_hub().switch()
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 294, in switch
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     return self.greenlet.switch()
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 136, in _run_loop
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     result = func(*self.args, **self.kw)
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 377, in _func
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     return self._sleep_time
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     self.force_reraise()
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     six.reraise(self.type_, self.value, self.tb)
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/oslo_service/loopingcall.py", line 356, in _func
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     result = f(*args, **kwargs)
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 342, in _do_wait_and_retry_detach
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]     reason=reason)
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6] DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain.)
  2016-03-24 16:34:44.921 143842 ERROR nova.compute.manager [instance: 88724ae0-38d6-4b06-9236-7d5f4d6d6cd6]

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1565859/+subscriptions
References

[Bug 1565859] [NEW] Can't detach volume from an instance
From: Lauren Taylor, 2016-04-04