yahoo-eng-team team mailing list archive

[Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>]

** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => kilo-1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1367918

Title:
  Xenapi attached volume with no VM leaves instance in undeletable state

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  As shown by the stack trace below, when a volume is attached but the
  VM is not present the volume can't be cleaned up by Cinder and will
  raise an Exception which puts the instance into an error state.  The
  volume attachment isn't removed because an if statement is hit in the
  xenapi destroy method which logs "VM is not present, skipping
  destroy..." and then moves on to trying to cleanup the volume in
  Cinder.  This is because most operations in xen rely on finding the
  vm_ref and then cleaning up resources that are attached there.  But if
  the volume is attached to an SR but not associated with an instance it
  ends up being orphaned.

  
  014-08-29 15:54:02.836 8766 DEBUG nova.volume.cinder [req-341cd17d-0f2f-4d64-929f-a94f8c0fa295 None] Cinderclient connection created using URL: https://localhost/v1/<tenant>
  cinderclient /opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/volume/cinder.py:108
  2014-08-29 15:54:03.251 8766 ERROR nova.compute.manager [req-341cd17d-0f2f-4d64-929f-a94f8c0fa295 None] [instance: <uuid>] Setting instance vm_state to ERROR
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] Traceback (most recent call last):
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2443, in do_terminate_instance
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] self._delete_instance(context, instance, bdms, quotas)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/hooks.py", line 131, in inner
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] rv = f(*args, **kwargs)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2412, in delete_instance
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] quotas.rollback()
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 82, in exit
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] six.reraise(self.type, self.value, self.tb)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2390, in _delete_instance
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] self._shutdown_instance(context, instance, bdms)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 2335, in _shutdown_instance
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] connector)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/volume/cinder.py", line 189, in wrapper
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] res = method(self, ctx, volume_id, *args, **kwargs)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/nova/volume/cinder.py", line 309, in terminate_connection
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] connector)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 331, in terminate_connection
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] {'connector': connector})
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/v1/volumes.py", line 250, in _action
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] return self.api.client.post(url, body=body)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/client.py", line 223, in post
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] return self._cs_request(url, 'POST', **kwargs)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/client.py", line 187, in _cs_request
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] **kwargs)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] File "/opt/rackstack/879.28/nova/lib/python2.6/site-packages/cinderclient/client.py", line 170, in request
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] raise exceptions.from_response(resp, body)
  2014-08-29 15:54:03.251 8766 TRACE nova.compute.manager [instance: <uuid>] ClientException: DELETE on http://localhost:8081/volumes/<volume_uuid>/export?force=False returned '409' with 'Volume '<volume_uuid>' is currently attached to '<ip>'' (HTTP 409) (Request-ID: req-d8a81cfc-5ba2-4bfb-b519-c92a2

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1367918/+subscriptions