yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1999125] Re: cinder retype volume raise exception with shutoff vm

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Launchpad Bug Tracker <1999125@xxxxxxxxxxxxxxxxxx>
Date: Sun, 09 Jul 2023 04:17:12 -0000
Reply-to: Bug 1999125 <1999125@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
[Expired for OpenStack Compute (nova) because there has been no activity
for 60 days.]

** Changed in: nova
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1999125

Title:
  cinder retype volume raise exception with shutoff vm

Status in Cinder:
  Incomplete
Status in OpenStack Compute (nova):
  Expired

Bug description:
  Environment:
  - openstack victoria version
  - iSCSI storage using Netapp driver

  Step to reproduce:
  1. Shutoff vm and wait for its complete
  2. Run the command cinder retype --migration-policy on-demand <vol-id> <new-type>
  3. Task will fail with Error log below

  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager [req-aea12cc1-b7f5-4c38-85d7-1c9094a1006a d3802c30ff814feaa3197d29e8a3a731 2c97f9ded4c843279d2de2c4a5c837
  01 - - -] Failed to copy volume f81a905d-05ea-4534-9b3b-a6762adf87f3 to 099ce4e2-8539-43dc-a2fa-8feca2845a3d: novaclient.exceptions.Conflict: Cannot 'swap_volume' instance a14b7344-3054-4d9a-9357-c0adc1b00650 while it is in vm_state stopped (HTTP 409) (Request-ID: req-7913c7fb-51d0-4a2a-8463-d8a9883dea0b)
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager Traceback (most recent call last):
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2292, in _migrate_volume_generic
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager     nova_api.update_server_volume(ctxt, instance_uuid,
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/compute/nova.py", line 186, in update_server_volume
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager     nova.volumes.update_server_volume(server_id,
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/api_versions.py", line 393, in substitution
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager     return methods[-1].func(obj, *args, **kwargs)
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/v2/volumes.py", line 124, in update_server_volume
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager     return self._update("/servers/%s/os-volume_attachments/%s" %
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/base.py", line 380, in _update
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager     resp, body = self.api.client.put(url, body=body)
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 404, in put
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager     return self.request(url, 'PUT', **kwargs)
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager   File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/client.py", line 78, in request
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager     raise exceptions.from_response(resp, body, url, method)
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager novaclient.exceptions.Conflict: Cannot 'swap_volume' instance a14b7344-3054-4d9a-9357-c0adc1b00650 whileit is in vm_state stopped (HTTP 409) (Request-ID: req-7913c7fb-51d0-4a2a-8463-d8a9883dea0b)
  2022-12-07 16:49:29.381 966 ERROR cinder.volume.manager
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server [req-aea12cc1-b7f5-4c38-85d7-1c9094a1006a d3802c30ff814feaa3197d29e8a3a731 2c97f9ded4c843279d2de2c4a5c83701 - - -] Exception during message handling: novaclient.exceptions.Conflict: Cannot 'swap_volume' instance a14b7344-3054-4d9a-9357-c0adc1b00650 while it is in vm_state stopped (HTTP 409) (Request-ID: req-7913c7fb-51d0-4a2a-8463-d8a9883dea0b)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309,in dispatch
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229,in _do_dispatch
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2957, in retype
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     _retype_error(context, volume, old_reservations,
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     raise value
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2953, in retype
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     self.migrate_volume(context, volume, host,
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2546, in migrate_volume
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     volume.save()
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     raise value
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2539, in migrate_volume
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     self._migrate_volume_generic(ctxt, volume, host, new_type_id)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2300, in _migrate_volume_generic
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     self._clean_temporary_volume(ctxt, volume,
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     raise value
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/volume/manager.py", line 2292, in _migrate_volume_generic
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     nova_api.update_server_volume(ctxt, instance_uuid,
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/cinder/compute/nova.py", line 186, in update_server_volume
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     nova.volumes.update_server_volume(server_id,
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/api_versions.py", line 393, in substitution
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     return methods[-1].func(obj, *args, **kwargs)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/v2/volumes.py", line 124, in update_server_volume
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     return self._update("/servers/%s/os-volume_attachments/%s" %
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/base.py", line 380, in _update
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.put(url, body=body)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 404, in put
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     return self.request(url, 'PUT', **kwargs)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python3.8/site-packages/novaclient/client.py", line 78, in request
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body, url, method)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server novaclient.exceptions.Conflict: Cannot 'swap_volume' instance a14b7344-3054-4d9a-9357-c0adc1b00650 while it is in vm_state stopped (HTTP 409) (Request-ID: req-7913c7fb-51d0-4a2a-8463-d8a9883dea0b)
  2022-12-07 16:49:29.408 966 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1999125/+subscriptions