yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1367918] [NEW] Xenapi attached volume with no VM leaves instance in undeletable state

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Andrew Laski <andrew.laski@xxxxxxxxxxxxx>
Date: Wed, 10 Sep 2014 20:44:53 -0000
Reply-to: Bug 1367918 <1367918@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

As shown by the stack trace below, when a volume is attached but the VM
is not present the volume can't be cleaned up by Cinder and will raise
an Exception which puts the instance into an error state.  The volume
attachment isn't removed because an if statement is hit in the xenapi
destroy method which logs "VM is not present, skipping destroy..." and
then moves on to trying to cleanup the volume in Cinder.  This is
because most operations in xen rely on finding the vm_ref and then
cleaning up resources that are attached there.  But if the volume is
attached to an SR but not associated with an instance it ends up being
orphaned.


014-08-29 15:54:02.836 8766 DEBUG nova.volume.cinder [req-341cd17d-0f2f-4d64-929f-a94f8c0fa295 None] Cinderclient connection created using URL: https://localhost/v1/<tenant>
cinderclient /opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/volume/cinder.py:108
2014-08-29 15:54:03.251 8766 ERROR nova.compute.manager [req-341cd17d-0f2f-4d64-929f-a94f8c0fa295 None] [instance: <uuid>] Setting instance vm_state to ERROR
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] Traceback (most recent call last):
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2443, in do_terminate_instance
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] self._delete_instance(context, instance, bdms, quotas)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/hooks.py", line 131, in inner
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] rv = f(*args, **kwargs)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2412, in delete_instance
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] quotas.rollback()
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 82, in exit
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] six.reraise(self.type, self.value, self.tb)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2390, in _delete_instance
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] self._shutdown_instance(context, instance, bdms)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2335, in _shutdown_instance
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] connector)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/volume/cinder.py", line 189, in wrapper
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] res = method(self, ctx, volume_id, *args, **kwargs)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/volume/cinder.py", line 309, in terminate_connection
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] connector)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 331, in terminate_connection
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] {'connector': connector})
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 250, in _action
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] return self.api.client.post(url, body=body)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/client.py", line 223, in post
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] return self._cs_request(url, 'POST', **kwargs)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/client.py", line 187, in _cs_request
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] **kwargs)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/client.py", line 170, in request
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] raise exceptions.from_response(resp, body)
2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] ClientException: DELETE on http://localhost:8081/volumes/<volume_uuid>/export?force=False returned '409' with 'Volume '<volume_uuid>' is currently attached to '<ip>'' (HTTP 409) (Request-ID: req-d8a81cfc-5ba2-4bfb-b519-c92a2

** Affects: nova
     Importance: Medium
     Assignee: Andrew Laski (alaski)
         Status: In Progress

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1367918

Title:
  Xenapi attached volume with no VM leaves instance in undeletable state

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  As shown by the stack trace below, when a volume is attached but the
  VM is not present the volume can't be cleaned up by Cinder and will
  raise an Exception which puts the instance into an error state.  The
  volume attachment isn't removed because an if statement is hit in the
  xenapi destroy method which logs "VM is not present, skipping
  destroy..." and then moves on to trying to cleanup the volume in
  Cinder.  This is because most operations in xen rely on finding the
  vm_ref and then cleaning up resources that are attached there.  But if
  the volume is attached to an SR but not associated with an instance it
  ends up being orphaned.

  
  014-08-29 15:54:02.836 8766 DEBUG nova.volume.cinder [req-341cd17d-0f2f-4d64-929f-a94f8c0fa295 None] Cinderclient connection created using URL: https://localhost/v1/<tenant>
  cinderclient /opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/volume/cinder.py:108
  2014-08-29 15:54:03.251 8766 ERROR nova.compute.manager [req-341cd17d-0f2f-4d64-929f-a94f8c0fa295 None] [instance: <uuid>] Setting instance vm_state to ERROR
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] Traceback (most recent call last):
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2443, in do_terminate_instance
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] self._delete_instance(context, instance, bdms, quotas)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/hooks.py", line 131, in inner
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] rv = f(*args, **kwargs)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2412, in delete_instance
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] quotas.rollback()
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 82, in exit
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] six.reraise(self.type, self.value, self.tb)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2390, in _delete_instance
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] self._shutdown_instance(context, instance, bdms)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2335, in _shutdown_instance
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] connector)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/volume/cinder.py", line 189, in wrapper
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] res = method(self, ctx, volume_id, *args, **kwargs)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/volume/cinder.py", line 309, in terminate_connection
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] connector)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 331, in terminate_connection
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] {'connector': connector})
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 250, in _action
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] return self.api.client.post(url, body=body)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/client.py", line 223, in post
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] return self._cs_request(url, 'POST', **kwargs)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/client.py", line 187, in _cs_request
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] **kwargs)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/client.py", line 170, in request
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] raise exceptions.from_response(resp, body)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] ClientException: DELETE on http://localhost:8081/volumes/<volume_uuid>/export?force=False returned '409' with 'Volume '<volume_uuid>' is currently attached to '<ip>'' (HTTP 409) (Request-ID: req-d8a81cfc-5ba2-4bfb-b519-c92a2

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1367918/+subscriptions
Follow ups

[Bug 1367918] Re: Xenapi attached volume with no VM leaves instance in undeletable state
From: Thierry Carrez, 2014-12-18
[Bug 1367918] [NEW] Xenapi attached volume with no VM leaves instance in undeletable state
From: Andrew Laski, 2014-09-10
References

[Bug 1367918] [NEW] Xenapi attached volume with no VM leaves instance in undeletable state
From: Andrew Laski, 2014-09-10