yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #87351
[Bug 1946350] [NEW] nova-live-migration evacuation failure due to slow lvchange -a command in c-vol during volume attachment update
Public bug reported:
https://zuul.opendev.org/t/openstack/build/62d2b102d32943dd90b5c487ddca37dc/log/job-
output.txt
2021-10-07 09:42:51.099622 | controller | + /opt/stack/nova/roles/run-evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:25 : nova evacuate evacuate-bfv-test
[..]
2021-10-07 09:44:58.467425 | controller | ++ /opt/stack/nova/roles/run-evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:37 : openstack server show evacuate-bfv-test -f value -c status
2021-10-07 09:45:01.300376 | controller | + /opt/stack/nova/roles/run-evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:37 : status=ERROR
https://zuul.opendev.org/t/openstack/build/62d2b102d32943dd90b5c487ddca37dc/log/controller/logs/screen-
n-cpu.txt
Oct 07 09:44:58.582574 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.volume.cinder [None req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c demo admin] Update attachment failed for attachment 7aa0e212-131f-4f2e-a418-94ca1ed7e952. Error: Unable to update the attachment. (HTTP 500) (Request-ID: req-e2c42f46-979b-49a4-87cc-6f63692ea526) Code: 500: cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-e2c42f46-979b-49a4-87cc-6f63692ea526)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [None req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c demo admin] [instance: a0029c88-4829-4159-88c5-193ec16fafc8] Instance failed block device setup: cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-e2c42f46-979b-49a4-87cc-6f63692ea526)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] Traceback (most recent call last):
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/compute/manager.py", line 3702, in _do_rebuild_instance
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] self.driver.rebuild(**kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/driver.py", line 339, in rebuild
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] raise NotImplementedError()
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] NotImplementedError
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8]
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] During handling of the above exception, another exception occurred:
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8]
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] Traceback (most recent call last):
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/compute/manager.py", line 1968, in _prep_block_device
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] driver_block_device.attach_block_devices(
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 860, in attach_block_devices
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] _log_and_attach(device)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 857, in _log_and_attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] bdm.attach(*attach_args, **attach_kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 759, in attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] super(DriverVolImageBlockDevice, self).attach(
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] ret_val = method(obj, context, *args, **kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 658, in attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] self._do_attach(context, instance, volume, volume_api,
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 643, in _do_attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] self._volume_attach(context, volume, connector, instance,
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 557, in _volume_attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] connection_info = volume_api.attachment_update(
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/volume/cinder.py", line 396, in wrapper
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] res = method(self, ctx, *args, **kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/volume/cinder.py", line 447, in wrapper
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] res = method(self, ctx, attachment_id, *args, **kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/volume/cinder.py", line 875, in attachment_update
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] LOG.error('Update attachment failed for attachment '
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] self.force_reraise()
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] raise self.value
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/volume/cinder.py", line 867, in attachment_update
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] attachment_ref = cinderclient(
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/api_versions.py", line 421, in substitution
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] return method.func(obj, *args, **kwargs)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/v3/attachments.py", line 75, in update
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] resp = self._update('/attachments/%s' % id, body)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/base.py", line 318, in _update
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] resp, body = self.api.client.put(url, body=body, **kwargs)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 230, in put
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] return self._cs_request(url, 'PUT', **kwargs)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 215, in _cs_request
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] return self.request(url, method, **kwargs)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 201, in request
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] raise exceptions.from_response(resp, body)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-e2c42f46-979b-49a4-87cc-6f63692ea526)
https://zuul.opendev.org/t/openstack/build/62d2b102d32943dd90b5c487ddca37dc/log/controller/logs/screen-c-api.txt
Oct 07 09:44:58.552516 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: DEBUG cinder.coordination [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo admin] Lock "b'cinder-attachment_update-795b38cf-f066-4df8-b3ae-738aa1bad2ef-ubuntu-focal-iweb-mtl01-0026833218'" released by "attachment_update" :: held 120.030s {{(pid=108509) _synchronized /opt/stack/cinder/cinder/coordination.py:193}}
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo admin] Unable to update the attachment.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 9f88a34a292b43089a974005bb325511
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments Traceback (most recent call last):
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 433, in get
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return self._queues[msg_id].get(block=True, timeout=timeout)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/lib/python3.8/queue.py", line 178, in get
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments raise Empty
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments _queue.Empty
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments During handling of the above exception, another exception occurred:
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments Traceback (most recent call last):
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/api/v3/attachments.py", line 250, in update
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments self.volume_api.attachment_update(context,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return caller(func, *(extras + args), **kw)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/coordination.py", line 186, in _synchronized
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return f(*a, **k)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/volume/api.py", line 2250, in attachment_update
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments self.volume_rpcapi.attachment_update(ctxt,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/rpc.py", line 198, in _wrapper
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return f(self, *args, **kwargs)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/volume/rpcapi.py", line 470, in attachment_update
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return cctxt.call(ctxt,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments result = self.transport._send(
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/transport.py", line 123, in _send
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return self._driver.send(target, ctxt, message,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 680, in send
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return self._send(target, ctxt, message, wait_for_reply, timeout,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 669, in _send
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments result = self._waiter.wait(msg_id, timeout,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments message = self.waiters.get(msg_id, timeout=timeout)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 435, in get
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments raise oslo_messaging.MessagingTimeout(
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 9f88a34a292b43089a974005bb325511
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments
https://zuul.opendev.org/t/openstack/build/62d2b102d32943dd90b5c487ddca37dc/log/compute1/logs/screen-
c-vol.txt
Oct 07 09:42:58.564774 ubuntu-focal-iweb-mtl01-0026833226 cinder-volume[76311]: DEBUG oslo_concurrency.processutils [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-795b38cf-f066-4df8-b3ae-738aa1bad2ef {{(pid=76311) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Oct 07 09:42:58.587336 ubuntu-focal-iweb-mtl01-0026833226 sudo[81649]: stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-795b38cf-f066-4df8-b3ae-738aa1bad2ef
Oct 07 09:42:58.587791 ubuntu-focal-iweb-mtl01-0026833226 sudo[81649]: pam_unix(sudo:session): session opened for user root by (uid=0)
[..]
Oct 07 09:44:59.045499 ubuntu-focal-iweb-mtl01-0026833226 cinder-volume[76311]: DEBUG oslo_concurrency.processutils [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-795b38cf-f066-4df8-b3ae-738aa1bad2ef" returned: 0 in 120.480s {{(pid=76311) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
[..]
Oct 07 09:45:02.795446 ubuntu-focal-iweb-mtl01-0026833226 cinder-volume[76311]: INFO cinder.volume.manager [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo None] attachment_update completed successfully.
** Affects: cinder
Importance: Undecided
Status: New
** Affects: nova
Importance: Undecided
Status: New
** Tags: gate-failure
** Also affects: nova
Importance: Undecided
Status: New
** Summary changed:
- Volume attachment update fails due to slow lvchange -a command in CI
+ nova-live-migration evacuation failure due to slow lvchange -a command in c-vol during volume attachment update
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1946350
Title:
nova-live-migration evacuation failure due to slow lvchange -a command
in c-vol during volume attachment update
Status in Cinder:
New
Status in OpenStack Compute (nova):
New
Bug description:
https://zuul.opendev.org/t/openstack/build/62d2b102d32943dd90b5c487ddca37dc/log/job-
output.txt
2021-10-07 09:42:51.099622 | controller | + /opt/stack/nova/roles/run-evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:25 : nova evacuate evacuate-bfv-test
[..]
2021-10-07 09:44:58.467425 | controller | ++ /opt/stack/nova/roles/run-evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:37 : openstack server show evacuate-bfv-test -f value -c status
2021-10-07 09:45:01.300376 | controller | + /opt/stack/nova/roles/run-evacuate-hook/files/test_evacuate.sh:evacuate_and_wait_for_active:37 : status=ERROR
https://zuul.opendev.org/t/openstack/build/62d2b102d32943dd90b5c487ddca37dc/log/controller/logs/screen-
n-cpu.txt
Oct 07 09:44:58.582574 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.volume.cinder [None req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c demo admin] Update attachment failed for attachment 7aa0e212-131f-4f2e-a418-94ca1ed7e952. Error: Unable to update the attachment. (HTTP 500) (Request-ID: req-e2c42f46-979b-49a4-87cc-6f63692ea526) Code: 500: cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-e2c42f46-979b-49a4-87cc-6f63692ea526)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [None req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c demo admin] [instance: a0029c88-4829-4159-88c5-193ec16fafc8] Instance failed block device setup: cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-e2c42f46-979b-49a4-87cc-6f63692ea526)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] Traceback (most recent call last):
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/compute/manager.py", line 3702, in _do_rebuild_instance
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] self.driver.rebuild(**kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/driver.py", line 339, in rebuild
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] raise NotImplementedError()
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] NotImplementedError
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8]
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] During handling of the above exception, another exception occurred:
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8]
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] Traceback (most recent call last):
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/compute/manager.py", line 1968, in _prep_block_device
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] driver_block_device.attach_block_devices(
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 860, in attach_block_devices
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] _log_and_attach(device)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 857, in _log_and_attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] bdm.attach(*attach_args, **attach_kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 759, in attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] super(DriverVolImageBlockDevice, self).attach(
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] ret_val = method(obj, context, *args, **kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 658, in attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] self._do_attach(context, instance, volume, volume_api,
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 643, in _do_attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] self._volume_attach(context, volume, connector, instance,
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/virt/block_device.py", line 557, in _volume_attach
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] connection_info = volume_api.attachment_update(
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/volume/cinder.py", line 396, in wrapper
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] res = method(self, ctx, *args, **kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/volume/cinder.py", line 447, in wrapper
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] res = method(self, ctx, attachment_id, *args, **kwargs)
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/volume/cinder.py", line 875, in attachment_update
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] LOG.error('Update attachment failed for attachment '
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] self.force_reraise()
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
Oct 07 09:44:58.620044 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] raise self.value
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/opt/stack/nova/nova/volume/cinder.py", line 867, in attachment_update
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] attachment_ref = cinderclient(
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/api_versions.py", line 421, in substitution
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] return method.func(obj, *args, **kwargs)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/v3/attachments.py", line 75, in update
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] resp = self._update('/attachments/%s' % id, body)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/base.py", line 318, in _update
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] resp, body = self.api.client.put(url, body=body, **kwargs)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 230, in put
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] return self._cs_request(url, 'PUT', **kwargs)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 215, in _cs_request
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] return self.request(url, method, **kwargs)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 201, in request
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] raise exceptions.from_response(resp, body)
Oct 07 09:44:58.628987 ubuntu-focal-iweb-mtl01-0026833218 nova-compute[107821]: ERROR nova.compute.manager [instance: a0029c88-4829-4159-88c5-193ec16fafc8] cinderclient.exceptions.ClientException: Unable to update the attachment. (HTTP 500) (Request-ID: req-e2c42f46-979b-49a4-87cc-6f63692ea526)
https://zuul.opendev.org/t/openstack/build/62d2b102d32943dd90b5c487ddca37dc/log/controller/logs/screen-c-api.txt
Oct 07 09:44:58.552516 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: DEBUG cinder.coordination [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo admin] Lock "b'cinder-attachment_update-795b38cf-f066-4df8-b3ae-738aa1bad2ef-ubuntu-focal-iweb-mtl01-0026833218'" released by "attachment_update" :: held 120.030s {{(pid=108509) _synchronized /opt/stack/cinder/cinder/coordination.py:193}}
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo admin] Unable to update the attachment.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 9f88a34a292b43089a974005bb325511
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments Traceback (most recent call last):
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 433, in get
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return self._queues[msg_id].get(block=True, timeout=timeout)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/lib/python3.8/queue.py", line 178, in get
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments raise Empty
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments _queue.Empty
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments During handling of the above exception, another exception occurred:
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments Traceback (most recent call last):
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/api/v3/attachments.py", line 250, in update
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments self.volume_api.attachment_update(context,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return caller(func, *(extras + args), **kw)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/coordination.py", line 186, in _synchronized
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return f(*a, **k)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/volume/api.py", line 2250, in attachment_update
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments self.volume_rpcapi.attachment_update(ctxt,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/rpc.py", line 198, in _wrapper
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return f(self, *args, **kwargs)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/opt/stack/cinder/cinder/volume/rpcapi.py", line 470, in attachment_update
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return cctxt.call(ctxt,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/client.py", line 189, in call
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments result = self.transport._send(
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/transport.py", line 123, in _send
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return self._driver.send(target, ctxt, message,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 680, in send
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments return self._send(target, ctxt, message, wait_for_reply, timeout,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 669, in _send
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments result = self._waiter.wait(msg_id, timeout,
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 559, in wait
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments message = self.waiters.get(msg_id, timeout=timeout)
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 435, in get
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments raise oslo_messaging.MessagingTimeout(
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID 9f88a34a292b43089a974005bb325511
Oct 07 09:44:58.573089 ubuntu-focal-iweb-mtl01-0026833218 devstack@c-api.service[108509]: ERROR cinder.api.v3.attachments
https://zuul.opendev.org/t/openstack/build/62d2b102d32943dd90b5c487ddca37dc/log/compute1/logs/screen-
c-vol.txt
Oct 07 09:42:58.564774 ubuntu-focal-iweb-mtl01-0026833226 cinder-volume[76311]: DEBUG oslo_concurrency.processutils [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo None] Running cmd (subprocess): sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-795b38cf-f066-4df8-b3ae-738aa1bad2ef {{(pid=76311) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:384}}
Oct 07 09:42:58.587336 ubuntu-focal-iweb-mtl01-0026833226 sudo[81649]: stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-795b38cf-f066-4df8-b3ae-738aa1bad2ef
Oct 07 09:42:58.587791 ubuntu-focal-iweb-mtl01-0026833226 sudo[81649]: pam_unix(sudo:session): session opened for user root by (uid=0)
[..]
Oct 07 09:44:59.045499 ubuntu-focal-iweb-mtl01-0026833226 cinder-volume[76311]: DEBUG oslo_concurrency.processutils [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-795b38cf-f066-4df8-b3ae-738aa1bad2ef" returned: 0 in 120.480s {{(pid=76311) execute /usr/local/lib/python3.8/dist-packages/oslo_concurrency/processutils.py:422}}
[..]
Oct 07 09:45:02.795446 ubuntu-focal-iweb-mtl01-0026833226 cinder-volume[76311]: INFO cinder.volume.manager [req-b1f53ae4-53a0-4705-bedc-f64276bfbc5c req-e2c42f46-979b-49a4-87cc-6f63692ea526 demo None] attachment_update completed successfully.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1946350/+subscriptions
