yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2077512] [NEW] resize doesnt validate state on volumes, instance set to error when attachment fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Tobias Urdin <2077512@xxxxxxxxxxxxxxxxxx>
Date: Wed, 21 Aug 2024 07:59:37 -0000
Reply-to: Bug 2077512 <2077512@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

nova 26.2.1 (zed) looks like same code path in master

If we issue a resize the _terminate_volume_connections() function in
nova/compute/manager.py will call cinder to delete the attachments, but
we never check the actual volume status before doing this so if the
volume(s) is in a invalid state we will ERROR (because we are in a
_error_out_instance_on_exception context) the instance and only an cloud
admin can restore it.

We should do either of:
* Validate volume status before doing a resize so that we don't fail (which could cause a race condition between check and actual attachment)
* We could handle the error and raise a InstanceFaultRollback and since we are in a _error_out_instance_on_exception context to reset the instance back to it's status, then instead set the error details on the instance or add a instance log entry

2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [None req-7b16c6f1-ccd8-46c8-a4c6-9b6a5b716233 833c476e9c1adb00f53b90cfaa50d077788946d8cd916606dee388eaaba2a71f 3b680fb64bc848be9b647895c8f89cdb - - 6ee763824d0145eeb1174bf30416df2d default] [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] Setting instance vm_state to ERROR: nova.exception.InvalidInput: Invalid input received: Invalid volume: Volume 33025874-2586-4687-ae9c-478d291a0381 status must be available or downloading to reserve, but the current status is backing-up. (HTTP 400) (Request-ID: req-029f5d6a-ac81-42c1-91ab-5e2dee69525e)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] Traceback (most recent call last):
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 397, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     res = method(self, ctx, *args, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 435, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     res = method(self, ctx, volume_id, *args, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 810, in attachment_create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     LOG.error('Create attachment failed for volume '
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     self.force_reraise()
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise self.value
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 802, in attachment_create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     attachment_ref = cinderclient(context, '3.44').attachments.create(
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/api_versions.py", line 421, in substitution
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return method.func(obj, *args, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/v3/attachments.py", line 39, in create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     retval = self._create('/attachments', body, 'attachment')
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/base.py", line 306, in _create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     resp, body = self.api.client.post(url, body=body)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 227, in post
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return self._cs_request(url, 'POST', **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 215, in _cs_request
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return self.request(url, method, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 201, in request
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise exceptions.from_response(resp, body)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] cinderclient.exceptions.BadRequest: Invalid volume: Volume 33025874-2586-4687-ae9c-478d291a0381 status must be available or downloading to reserve, but the current status is backing-up. (HTTP 400) (Request-ID: req-029f5d6a-ac81-42c1-91ab-5e2dee69525e)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] 
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] During handling of the above exception, another exception occurred:
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] 
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] Traceback (most recent call last):
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 10612, in _error_out_instance_on_exception
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     yield
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5896, in _resize_instance
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     self._terminate_volume_connections(context, instance, bdms)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5964, in _terminate_volume_connections
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     attachment_id = self.volume_api.attachment_create(
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 405, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     _reraise(exception.InvalidInput(reason=err_msg))
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 488, in _reraise
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise desired_exc.with_traceback(sys.exc_info()[2])
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 397, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     res = method(self, ctx, *args, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 435, in wrapper
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     res = method(self, ctx, volume_id, *args, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 810, in attachment_create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     LOG.error('Create attachment failed for volume '
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     self.force_reraise()
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise self.value
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 802, in attachment_create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     attachment_ref = cinderclient(context, '3.44').attachments.create(
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/api_versions.py", line 421, in substitution
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return method.func(obj, *args, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/v3/attachments.py", line 39, in create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     retval = self._create('/attachments', body, 'attachment')
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/base.py", line 306, in _create
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     resp, body = self.api.client.post(url, body=body)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 227, in post
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return self._cs_request(url, 'POST', **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 215, in _cs_request
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return self.request(url, method, **kwargs)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 201, in request
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise exceptions.from_response(resp, body)
2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] nova.exception.InvalidInput: Invalid input received: Invalid volume: Volume 33025874-2586-4687-ae9c-478d291a0381 status must be available or downloading to reserve, but the current status is backing-up. (HTTP 400) (Request-ID: req-029f5d6a-ac81-42c1-91ab-5e2dee69525e)

** Affects: nova
     Importance: Undecided
         Status: New

** Summary changed:

- resize doesnt validate state on volumes before running
+ resize doesnt validate state on volumes, instance set to error when attachment fails

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2077512

Title:
  resize doesnt validate state on volumes, instance set to error when
  attachment fails

Status in OpenStack Compute (nova):
  New

Bug description:
  nova 26.2.1 (zed) looks like same code path in master

  If we issue a resize the _terminate_volume_connections() function in
  nova/compute/manager.py will call cinder to delete the attachments,
  but we never check the actual volume status before doing this so if
  the volume(s) is in a invalid state we will ERROR (because we are in a
  _error_out_instance_on_exception context) the instance and only an
  cloud admin can restore it.

  We should do either of:
  * Validate volume status before doing a resize so that we don't fail (which could cause a race condition between check and actual attachment)
  * We could handle the error and raise a InstanceFaultRollback and since we are in a _error_out_instance_on_exception context to reset the instance back to it's status, then instead set the error details on the instance or add a instance log entry

  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [None req-7b16c6f1-ccd8-46c8-a4c6-9b6a5b716233 833c476e9c1adb00f53b90cfaa50d077788946d8cd916606dee388eaaba2a71f 3b680fb64bc848be9b647895c8f89cdb - - 6ee763824d0145eeb1174bf30416df2d default] [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] Setting instance vm_state to ERROR: nova.exception.InvalidInput: Invalid input received: Invalid volume: Volume 33025874-2586-4687-ae9c-478d291a0381 status must be available or downloading to reserve, but the current status is backing-up. (HTTP 400) (Request-ID: req-029f5d6a-ac81-42c1-91ab-5e2dee69525e)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] Traceback (most recent call last):
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 397, in wrapper
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     res = method(self, ctx, *args, **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 435, in wrapper
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     res = method(self, ctx, volume_id, *args, **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 810, in attachment_create
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     LOG.error('Create attachment failed for volume '
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     self.force_reraise()
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise self.value
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 802, in attachment_create
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     attachment_ref = cinderclient(context, '3.44').attachments.create(
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/api_versions.py", line 421, in substitution
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return method.func(obj, *args, **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/v3/attachments.py", line 39, in create
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     retval = self._create('/attachments', body, 'attachment')
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/base.py", line 306, in _create
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     resp, body = self.api.client.post(url, body=body)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 227, in post
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return self._cs_request(url, 'POST', **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 215, in _cs_request
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return self.request(url, method, **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 201, in request
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise exceptions.from_response(resp, body)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] cinderclient.exceptions.BadRequest: Invalid volume: Volume 33025874-2586-4687-ae9c-478d291a0381 status must be available or downloading to reserve, but the current status is backing-up. (HTTP 400) (Request-ID: req-029f5d6a-ac81-42c1-91ab-5e2dee69525e)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] 
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] During handling of the above exception, another exception occurred:
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] 
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] Traceback (most recent call last):
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 10612, in _error_out_instance_on_exception
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     yield
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5896, in _resize_instance
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     self._terminate_volume_connections(context, instance, bdms)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 5964, in _terminate_volume_connections
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     attachment_id = self.volume_api.attachment_create(
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 405, in wrapper
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     _reraise(exception.InvalidInput(reason=err_msg))
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 488, in _reraise
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise desired_exc.with_traceback(sys.exc_info()[2])
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 397, in wrapper
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     res = method(self, ctx, *args, **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 435, in wrapper
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     res = method(self, ctx, volume_id, *args, **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 810, in attachment_create
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     LOG.error('Create attachment failed for volume '
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     self.force_reraise()
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise self.value
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/nova/volume/cinder.py", line 802, in attachment_create
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     attachment_ref = cinderclient(context, '3.44').attachments.create(
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/api_versions.py", line 421, in substitution
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return method.func(obj, *args, **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/v3/attachments.py", line 39, in create
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     retval = self._create('/attachments', body, 'attachment')
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/base.py", line 306, in _create
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     resp, body = self.api.client.post(url, body=body)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 227, in post
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return self._cs_request(url, 'POST', **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 215, in _cs_request
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     return self.request(url, method, **kwargs)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]   File "/usr/lib/python3.9/site-packages/cinderclient/client.py", line 201, in request
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8]     raise exceptions.from_response(resp, body)
  2024-08-20 11:27:43.868 2080485 ERROR nova.compute.manager [instance: a7fae912-b312-4d3e-9963-4d3b06f705f8] nova.exception.InvalidInput: Invalid input received: Invalid volume: Volume 33025874-2586-4687-ae9c-478d291a0381 status must be available or downloading to reserve, but the current status is backing-up. (HTTP 400) (Request-ID: req-029f5d6a-ac81-42c1-91ab-5e2dee69525e)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2077512/+subscriptions