yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1515637] [NEW] Double detach volume causes server fault

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Marian Horban <mhorban@xxxxxxxxxxxx>
Date: Thu, 12 Nov 2015 14:14:09 -0000
Reply-to: Bug 1515637 <1515637@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

If volume in 'detaching'  state and detach operation is called nova-api
fails:

2015-11-10 05:18:19.253 ERROR nova.api.openstack.extensions [req-05889195-e70d-4761-a5c6-a69ddfe05d62 tempest-ServerActionsTestJSON-653602906 tempest-ServerActionsTestJSON-743378399] Unexpected exception in API method
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions Traceback (most recent call last):
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 478, in wrapped
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/volumes.py", line 395, in delete
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     self.compute_api.detach_volume(context, instance, volume)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 235, in wrapped
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return func(self, context, target, *args, **kwargs)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 224, in inner
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 205, in inner
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 3098, in detach_volume
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     self._detach_volume(context, instance, volume)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 3080, in _detach_volume
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     self.volume_api.begin_detaching(context, volume['id'])
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/volume/cinder.py", line 235, in wrapper
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     six.reraise(exc_value, None, exc_trace)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/volume/cinder.py", line 224, in wrapper
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     res = method(self, ctx, volume_id, *args, **kwargs)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/volume/cinder.py", line 335, in begin_detaching
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     cinderclient(context).volumes.begin_detaching(volume_id)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 454, in begin_detaching
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return self._action('os-begin_detaching', volume)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 402, in _action
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return self.api.client.post(url, body=body)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 104, in post
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return self._cs_request(url, 'POST', **kwargs)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 98, in _cs_request
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return self.request(url, method, **kwargs)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 91, in request
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     raise exceptions.from_response(resp, body)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions InvalidInput: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. Currently: status: 'detaching', attach_status: 'attached.' (HTTP 400) (Request-ID: req-f91e3713-7538-4285-af29-bfa7dbdbb2ab)
2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions

It could be easily reproduced with two consecutive detach volume
operations.

** Affects: nova
     Importance: Undecided
     Assignee: Marian Horban (mhorban)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Marian Horban (mhorban)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1515637

Title:
  Double detach volume causes server fault

Status in OpenStack Compute (nova):
  New

Bug description:
  If volume in 'detaching'  state and detach operation is called nova-
  api fails:

  2015-11-10 05:18:19.253 ERROR nova.api.openstack.extensions [req-05889195-e70d-4761-a5c6-a69ddfe05d62 tempest-ServerActionsTestJSON-653602906 tempest-ServerActionsTestJSON-743378399] Unexpected exception in API method
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions Traceback (most recent call last):
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 478, in wrapped
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return f(*args, **kwargs)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/volumes.py", line 395, in delete
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     self.compute_api.detach_volume(context, instance, volume)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 235, in wrapped
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return func(self, context, target, *args, **kwargs)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 224, in inner
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return function(self, context, instance, *args, **kwargs)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 205, in inner
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return f(self, context, instance, *args, **kw)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 3098, in detach_volume
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     self._detach_volume(context, instance, volume)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 3080, in _detach_volume
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     self.volume_api.begin_detaching(context, volume['id'])
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/volume/cinder.py", line 235, in wrapper
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     six.reraise(exc_value, None, exc_trace)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/volume/cinder.py", line 224, in wrapper
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     res = method(self, ctx, volume_id, *args, **kwargs)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/opt/stack/nova/nova/volume/cinder.py", line 335, in begin_detaching
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     cinderclient(context).volumes.begin_detaching(volume_id)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 454, in begin_detaching
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return self._action('os-begin_detaching', volume)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/v2/volumes.py", line 402, in _action
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return self.api.client.post(url, body=body)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 104, in post
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return self._cs_request(url, 'POST', **kwargs)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 98, in _cs_request
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     return self.request(url, method, **kwargs)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/cinderclient/client.py", line 91, in request
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions     raise exceptions.from_response(resp, body)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions InvalidInput: Invalid input received: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. Currently: status: 'detaching', attach_status: 'attached.' (HTTP 400) (Request-ID: req-f91e3713-7538-4285-af29-bfa7dbdbb2ab)
  2015-11-10 05:18:19.253 TRACE nova.api.openstack.extensions

  It could be easily reproduced with two consecutive detach volume
  operations.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1515637/+subscriptions
Follow ups

[Bug 1515637] Re: Double detach volume causes server fault
From: Markus Zoeller (markus_z), 2015-12-09