yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1888665] [NEW] Cinder volume stuck in detaching on error exception

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jacolex <1888665@xxxxxxxxxxxxxxxxxx>
Date: Thu, 23 Jul 2020 12:51:36 -0000
Reply-to: Bug 1888665 <1888665@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hello!

The issue is annoying me and my users for long time, so I decided to report the problem with detaching volumes from instances. When an exception occurs, the detaching is stuck in detaching state without retrying detach procedure.
For example, below is the exception:

2020-07-23 09:36:37.964 37220 ERROR nova.volume.cinder [req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Delete attachment failed for attachment 35653c6c-12c5-4d10-a3d4-508b45e2b1d6. Error: Gat
eway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server [req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Exception during message handling: cinderclient.exceptions.ClientException: Gatew
ay Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 79, in wrapped
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1372, in decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 219, in decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in decorated_function
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6198, in detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     do_detach_volume(context, volume_id, instance, attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6196, in do_detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6147, in _detach_volume
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id, destroy_bdm=destroy_bdm)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 461, in detach
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id, destroy_bdm)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 442, in _do_detach
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     volume_api.attachment_delete(context, self['attachment_id'])
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 397, in wrapper
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 448, in wrapper
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 889, in attachment_delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     'code': getattr(ex, 'code', None)})
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     self.force_reraise()
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise value
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 882, in attachment_delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/api_versions.py", line 423, in substitution
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return method.func(obj, *args, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/v3/attachments.py", line 44, in delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/base.py", line 309, in _delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 223, in delete
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 205, in _cs_request
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 191, in request
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server

After that the volume can't be detached without reset-state to "in-use".
I suggest to include some timeout or retrying in attach/detach
procedures.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: cinder nova-compute

** Description changed:

  Hello!
  
- The issue is annoying me and my users for long time, so I decided to reported the problem with detaching volumes from instances. When exception occurs, detaching is stuck in detaching state without retrying detach procedure.
+ The issue is annoying me and my users for long time, so I decided to report the problem with detaching volumes from instances. When an exception occurs, the detaching is stuck in detaching state without retrying detach procedure.
  For example, below is the exception:
  
  2020-07-23 09:36:37.964 37220 ERROR nova.volume.cinder [req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Delete attachment failed for attachment 35653c6c-12c5-4d10-a3d4-508b45e2b1d6. Error: Gat
  eway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server [req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Exception during message handling: cinderclient.exceptions.ClientException: Gatew
  ay Time-out (HTTP 504)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 79, in wrapped
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise value
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1372, in decorated_function
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 219, in decorated_function
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise value
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in decorated_function
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6198, in detach_volume
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     do_detach_volume(context, volume_id, instance, attachment_id)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6196, in do_detach_volume
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6147, in _detach_volume
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id, destroy_bdm=destroy_bdm)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 461, in detach
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id, destroy_bdm)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 442, in _do_detach
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     volume_api.attachment_delete(context, self['attachment_id'])
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 397, in wrapper
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 448, in wrapper
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 889, in attachment_delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     'code': getattr(ex, 'code', None)})
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise value
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 882, in attachment_delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/api_versions.py", line 423, in substitution
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return method.func(obj, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/v3/attachments.py", line 44, in delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/base.py", line 309, in _delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 223, in delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 205, in _cs_request
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 191, in request
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
- 2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server 
+ 2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server
  
- 
- After that the volume can't be detached without reset-state to "in-use". I suggest to include some timeout or retrying in attach/detach procedures.
+ After that the volume can't be detached without reset-state to "in-use".
+ I suggest to include some timeout or retrying in attach/detach
+ procedures.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1888665

Title:
  Cinder volume stuck in detaching on error exception

Status in OpenStack Compute (nova):
  New

Bug description:
  Hello!

  The issue is annoying me and my users for long time, so I decided to report the problem with detaching volumes from instances. When an exception occurs, the detaching is stuck in detaching state without retrying detach procedure.
  For example, below is the exception:

  2020-07-23 09:36:37.964 37220 ERROR nova.volume.cinder [req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Delete attachment failed for attachment 35653c6c-12c5-4d10-a3d4-508b45e2b1d6. Error: Gat
  eway Time-out (HTTP 504) Code: 504: cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server [req-1397b7e6-e111-44e6-87df-3057b30bfa0b 13622fba11a84996a70019d553060c8c b5fec10eaaa948ee86d61d7db3fb42e7 - default default] Exception during message handling: cinderclient.exceptions.ClientException: Gatew
  ay Time-out (HTTP 504)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 79, in wrapped
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise value
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 69, in wrapped
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1372, in decorated_function
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 219, in decorated_function
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise value
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 207, in decorated_function
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6198, in detach_volume
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     do_detach_volume(context, volume_id, instance, attachment_id)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 328, in inner
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6196, in do_detach_volume
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 6147, in _detach_volume
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id, destroy_bdm=destroy_bdm)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 461, in detach
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id, destroy_bdm)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/block_device.py", line 442, in _do_detach
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     volume_api.attachment_delete(context, self['attachment_id'])
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 397, in wrapper
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 448, in wrapper
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 889, in attachment_delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     'code': getattr(ex, 'code', None)})
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise value
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/volume/cinder.py", line 882, in attachment_delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     attachment_id)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/api_versions.py", line 423, in substitution
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return method.func(obj, *args, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/v3/attachments.py", line 44, in delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/base.py", line 309, in _delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 223, in delete
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 205, in _cs_request
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/cinderclient/client.py", line 191, in request
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server cinderclient.exceptions.ClientException: Gateway Time-out (HTTP 504)
  2020-07-23 09:36:38.019 37220 ERROR oslo_messaging.rpc.server

  After that the volume can't be detached without reset-state to "in-
  use". I suggest to include some timeout or retrying in attach/detach
  procedures.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1888665/+subscriptions