yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2089519] [NEW] instance evacuate faile due to it's volume have two attachments

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: benlei <2089519@xxxxxxxxxxxxxxxxxx>
Date: Mon, 25 Nov 2024 06:12:44 -0000
Reply-to: Bug 2089519 <2089519@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

In the _post_live_migration_remove_source_vol_connections phase of live
migration, if the attachment delete failed and it will not raise
exception, but the instance's volume attachment that based source node
will remain and instance will create a new attachment record on the
destination node. So, the instance' volume will have two attachments.

And now, if we execute instance evacuation action, instance evacuate
will failed due to it's volume have two attachments.

nova raise exception as below:

2024-10-30 06:20:12.381 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.381471453+08:00 stdout F 2024-10-30 06:20:12.380 25474 ERROR nova.volume.cinder [req-43f85cdc-e536-4304-b384-663c54aaced4 6c0e93adbe254ed3b8a34c93ebdf3b32 0e74a3f170944594ab1c4c6ca7dd5953 - default default] Update attachment failed for attachment f309ae52-3954-47b3-a43f-fed9f426853b. Error: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID: req-5c35c1cf-f2bb-4d03-95d0-86e952061794) Code: 500: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID: req-5c35c1cf-f2bb-4d03-95d0-86e952061794)ESC[00m
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751598438+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [req-43f85cdc-e536-4304-b384-663c54aaced4 6c0e93adbe254ed3b8a34c93ebdf3b32 0e74a3f170944594ab1c4c6ca7dd5953 - default default] [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] Instance failed block device setup: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID: req-5c35c1cf-f2bb-4d03-95d0-86e952061794)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751684297+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] Traceback (most recent call last):
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751704177+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 3765, in _do_rebuild_instance
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751782996+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     self.driver.rebuild(**kwargs)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751796616+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/driver.py", line 337, in rebuild
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751806716+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     raise NotImplementedError()
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751815136+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] NotImplementedError
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751822956+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751834196+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] During handling of the above exception, another exception occurred:
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751841496+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751848456+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] Traceback (most recent call last):
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751854936+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 1992, in _prep_block_device
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751880296+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     wait_func=self._await_block_device_map_created)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751888656+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 893, in attach_block_devices
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751895956+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     _log_and_attach(device)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751903316+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 890, in _log_and_attach
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751909876+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     bdm.attach(*attach_args, **attach_kwargs)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751916336+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 789, in attach
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751923896+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     context, instance, volume_api, virt_driver)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751930455+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 50, in wrapped
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751937435+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     ret_val = method(obj, context, *args, **kwargs)
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751943835+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 677, in attach
2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751990755+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     virt_driver, do_driver_attach)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752007575+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 664, in _do_attach
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752014735+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     do_driver_attach)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752022115+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 577, in _volume_attach
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752029415+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     self['mount_device'])['connection_info']
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752035995+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 397, in wrapper
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752042835+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     res = method(self, ctx, *args, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752049595+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 448, in wrapper
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752056235+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     res = method(self, ctx, attachment_id, *args, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752063055+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 908, in attachment_update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752069575+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     'code': getattr(ex, 'code', None)})
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752076374+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752108214+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     self.force_reraise()
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752117074+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752123894+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     raise self.value
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752130414+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 898, in attachment_update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752137574+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     attachment_id, _connector)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752145134+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/api_versions.py", line 423, in substitution
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752151634+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     return method.func(obj, *args, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752158154+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 75, in update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752166374+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     resp = self._update('/attachments/%s' % id, body)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752173034+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/base.py", line 312, in _update
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752181094+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     resp, body = self.api.client.put(url, body=body, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752221713+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 220, in put
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752272273+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     return self._cs_request(url, 'PUT', **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752283493+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 205, in _cs_request
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752290613+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     return self.request(url, method, **kwargs)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752297433+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 191, in request
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752304133+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     raise exceptions.from_response(resp, body)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752316893+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID: req-5c35c1cf-f2bb-4d03-95d0-86e952061794)
2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752325713+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] ESC[00m

Do we have any solution for this scenario?

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2089519

Title:
  instance evacuate faile due to it's volume have two attachments

Status in OpenStack Compute (nova):
  New

Bug description:
  In the _post_live_migration_remove_source_vol_connections phase of
  live migration, if the attachment delete failed and it will not raise
  exception, but the instance's volume attachment that based source node
  will remain and instance will create a new attachment record on the
  destination node. So, the instance' volume will have two attachments.

  And now, if we execute instance evacuation action, instance evacuate
  will failed due to it's volume have two attachments.

  nova raise exception as below:

  2024-10-30 06:20:12.381 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.381471453+08:00 stdout F 2024-10-30 06:20:12.380 25474 ERROR nova.volume.cinder [req-43f85cdc-e536-4304-b384-663c54aaced4 6c0e93adbe254ed3b8a34c93ebdf3b32 0e74a3f170944594ab1c4c6ca7dd5953 - default default] Update attachment failed for attachment f309ae52-3954-47b3-a43f-fed9f426853b. Error: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID: req-5c35c1cf-f2bb-4d03-95d0-86e952061794) Code: 500: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID: req-5c35c1cf-f2bb-4d03-95d0-86e952061794)ESC[00m
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751598438+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [req-43f85cdc-e536-4304-b384-663c54aaced4 6c0e93adbe254ed3b8a34c93ebdf3b32 0e74a3f170944594ab1c4c6ca7dd5953 - default default] [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] Instance failed block device setup: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID: req-5c35c1cf-f2bb-4d03-95d0-86e952061794)
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751684297+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] Traceback (most recent call last):
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751704177+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 3765, in _do_rebuild_instance
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751782996+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     self.driver.rebuild(**kwargs)
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751796616+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/driver.py", line 337, in rebuild
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751806716+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     raise NotImplementedError()
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751815136+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] NotImplementedError
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751822956+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751834196+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] During handling of the above exception, another exception occurred:
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751841496+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751848456+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] Traceback (most recent call last):
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751854936+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/compute/manager.py", line 1992, in _prep_block_device
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751880296+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     wait_func=self._await_block_device_map_created)
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751888656+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 893, in attach_block_devices
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751895956+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     _log_and_attach(device)
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751903316+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 890, in _log_and_attach
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751909876+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     bdm.attach(*attach_args, **attach_kwargs)
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751916336+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 789, in attach
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751923896+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     context, instance, volume_api, virt_driver)
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751930455+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 50, in wrapped
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751937435+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     ret_val = method(obj, context, *args, **kwargs)
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751943835+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 677, in attach
  2024-10-30 06:20:12.751 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.751990755+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     virt_driver, do_driver_attach)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752007575+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 664, in _do_attach
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752014735+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     do_driver_attach)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752022115+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/virt/block_device.py", line 577, in _volume_attach
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752029415+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     self['mount_device'])['connection_info']
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752035995+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 397, in wrapper
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752042835+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     res = method(self, ctx, *args, **kwargs)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752049595+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 448, in wrapper
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752056235+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     res = method(self, ctx, attachment_id, *args, **kwargs)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752063055+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 908, in attachment_update
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752069575+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     'code': getattr(ex, 'code', None)})
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752076374+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752108214+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     self.force_reraise()
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752117074+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752123894+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     raise self.value
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752130414+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/nova/volume/cinder.py", line 898, in attachment_update
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752137574+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     attachment_id, _connector)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752145134+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/api_versions.py", line 423, in substitution
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752151634+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     return method.func(obj, *args, **kwargs)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752158154+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/v3/attachments.py", line 75, in update
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752166374+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     resp = self._update('/attachments/%s' % id, body)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752173034+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/base.py", line 312, in _update
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752181094+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     resp, body = self.api.client.put(url, body=body, **kwargs)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752221713+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 220, in put
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752272273+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     return self._cs_request(url, 'PUT', **kwargs)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752283493+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 205, in _cs_request
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752290613+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     return self.request(url, method, **kwargs)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752297433+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]   File "/usr/local/lib/python3.6/site-packages/cinderclient/client.py", line 191, in request
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752304133+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f]     raise exceptions.from_response(resp, body)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752316893+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 968397d1-ab95-4637-88b5-e1fcc464404b). (HTTP 500) (Request-ID: req-5c35c1cf-f2bb-4d03-95d0-86e952061794)
  2024-10-30 06:20:12.752 +0800 ¦ node-140 ¦ nova-compute-n5frs ¦ nova-compute ¦ 2024-10-30T06:20:12.752325713+08:00 stdout F 2024-10-30 06:20:12.413 25474 ERROR nova.compute.manager [instance: 4deeed7a-9e86-4e6d-a95d-730e1c26629f] ESC[00m

  Do we have any solution for this scenario?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2089519/+subscriptions