yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1432984] [NEW] volume hangs in "detaching" state after cinder "reset-state" was used

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thomas Bechtold <tbechtold@xxxxxxxx>
Date: Tue, 17 Mar 2015 08:09:15 -0000
Reply-to: Bug 1432984 <1432984@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

With Ceph as cinder backend, detaching a volume doesn't work after the
following steps:

# boot volume, create instance
nova boot --poll --key-name testkey --flavor 2 --image c848067a-a8bb-4631-a6a6-ed0404d444a9 test01
cinder create 1
export INSTANCEID=adcfcda6-5a27-4c85-a770-210a6b08894b
export VOLID=fad11ab9-3a1f-43d7-83c3-55bded6280e4
nova volume-attach $INSTANCEID $VOLID
sleep 1
cinder reset-state --state available $VOLID
sleep 1
nova volume-detach $INSTANCEID $VOLID
sleep 1
nova volume-attach $INSTANCEID $VOLID
sleep 1
cinder reset-state --state in-use $VOLID
sleep 1
nova volume-detach $INSTANCEID $VOLID


After the last step, the volume hangs in "detaching" state. nova-compute.log shows:

2015-03-17 08:06:06.456 1342 AUDIT nova.compute.manager [req-68662304-4140-4e19-b4dc-ec65cb43ec2b 04030aff6a0a4416bc532863109343ff 362d7cfe3e7b491290f7d1684db0a48f] [instance: adcfcda6-5a27-4c85-a770-210a6b08894b] Detach volume fad11ab9-3a1f-43d7-83c3-55bded6280e4 from mountpoint /dev/vdb
2015-03-17 08:06:06.574 1342 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: expected string or buffer
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     payload)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     pass
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267, in detach_volume
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     self._detach_volume(context, instance, bdm)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4212, in _detach_volume
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     connection_info = jsonutils.loads(bdm.connection_info)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line 164, in loads
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return json.loads(s)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return _default_decoder.decode(s)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher TypeError: expected string or buffer
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher 
2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-] Returning exception expected string or buffer to caller
2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 88, in wrapped\n    payload)\n', '  File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function\n    pass\n', '  File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in decorated_function\n    e, sys.exc_info())\n', '  File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267, in detach_volume\n    self._detach_volume(context, instance, bdm)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4212, in _detach_volume\n    connection_info = jsonutils.loads(bdm.connection_info)\n', '  File "/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line 164, in loads\n    return json.loads(s)\n', '  File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads\n    return _default_decoder.decode(s)\n', '  File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n', 'TypeError: expected string or buffer\n']


The problem seems that the entry in the block_device_mapping table is
removed for the attached volume which leads to this error.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: ceph volumes

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1432984

Title:
  volume hangs in "detaching" state after cinder "reset-state" was used

Status in OpenStack Compute (Nova):
  New

Bug description:
  With Ceph as cinder backend, detaching a volume doesn't work after the
  following steps:

  # boot volume, create instance
  nova boot --poll --key-name testkey --flavor 2 --image c848067a-a8bb-4631-a6a6-ed0404d444a9 test01
  cinder create 1
  export INSTANCEID=adcfcda6-5a27-4c85-a770-210a6b08894b
  export VOLID=fad11ab9-3a1f-43d7-83c3-55bded6280e4
  nova volume-attach $INSTANCEID $VOLID
  sleep 1
  cinder reset-state --state available $VOLID
  sleep 1
  nova volume-detach $INSTANCEID $VOLID
  sleep 1
  nova volume-attach $INSTANCEID $VOLID
  sleep 1
  cinder reset-state --state in-use $VOLID
  sleep 1
  nova volume-detach $INSTANCEID $VOLID

  
  After the last step, the volume hangs in "detaching" state. nova-compute.log shows:

  2015-03-17 08:06:06.456 1342 AUDIT nova.compute.manager [req-68662304-4140-4e19-b4dc-ec65cb43ec2b 04030aff6a0a4416bc532863109343ff 362d7cfe3e7b491290f7d1684db0a48f] [instance: adcfcda6-5a27-4c85-a770-210a6b08894b] Detach volume fad11ab9-3a1f-43d7-83c3-55bded6280e4 from mountpoint /dev/vdb
  2015-03-17 08:06:06.574 1342 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: expected string or buffer
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 88, in wrapped
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     payload)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     pass
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in decorated_function
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in decorated_function
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267, in detach_volume
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     self._detach_volume(context, instance, bdm)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4212, in _detach_volume
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     connection_info = jsonutils.loads(bdm.connection_info)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line 164, in loads
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return json.loads(s)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     return _default_decoder.decode(s)
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher TypeError: expected string or buffer
  2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-] Returning exception expected string or buffer to caller
  2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 88, in wrapped\n    payload)\n', '  File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function\n    pass\n', '  File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in decorated_function\n    e, sys.exc_info())\n', '  File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267, in detach_volume\n    self._detach_volume(context, instance, bdm)\n', '  File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4212, in _detach_volume\n    connection_info = jsonutils.loads(bdm.connection_info)\n', '  File "/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line 164, in loads\n    return json.loads(s)\n', '  File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads\n    return _default_decoder.decode(s)\n', '  File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n', 'TypeError: expected string or buffer\n']


  The problem seems that the entry in the block_device_mapping table is
  removed for the attached volume which leads to this error.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1432984/+subscriptions
Follow ups

[Bug 1432984] Re: volume hangs in "detaching" state after cinder "reset-state" was used
From: Thomas Bechtold, 2015-03-17
[Bug 1432984] [NEW] volume hangs in "detaching" state after cinder "reset-state" was used
From: Thomas Bechtold, 2015-03-17
References

[Bug 1432984] [NEW] volume hangs in "detaching" state after cinder "reset-state" was used
From: Thomas Bechtold, 2015-03-17