← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #86313

[Bug 1931716] [NEW] Detaching volume from the live domain: virDomainDetachDeviceFlags(): libvirt.libvirtError: operation failed: disk vdb not found

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========

The test_attach_attached_volume_to_same_server tests within the nova-
next job fail during detaching the volume:

File "/usr/local/lib/python3.8/dist-packages/libvirt.py", line 1515, in
detachDeviceFlags

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     raise
libvirtError('virDomainDetachDeviceFlags() failed')

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5] libvirt.libvirtError: operation
failed: disk vdb not found


Steps to reproduce
==================

Only seen as part of the nova-next job at present.

Expected result
===============

detach succeeds

Actual result
=============

test fails as the volume state

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

master

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

libvirt

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

cinder volume

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

N/A

Logs & Configs
==============

https://zuul.opendev.org/t/openstack/build/02e6a99bf1574c978c663eb434705cbb/log/controller/logs/screen-n-cpu.txt?severity=0#34811

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [None req-aceedc3b-f03b-
4bd5-8e8e-cfe66170a2e2 tempest-AttachVolumeNegativeTest-1273499975
tempest-AttachVolumeNegativeTest-1273499975-project] [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5] Failed to detach volume
2939aa92-0156-45d6-9689-ed48dcc8fd8a from /dev/vdb:
libvirt.libvirtError: operation failed: disk vdb not found

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5] Traceback (most recent call last):

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
"/opt/stack/nova/nova/virt/block_device.py", line 328, in driver_detach

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
virt_driver.detach_volume(context, connection_info, instance, mp,

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2555, in
detach_volume

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     self._detach_with_retry(

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2308, in
_detach_with_retry

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
self._detach_from_live_with_retry(

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2364, in
_detach_from_live_with_retry

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
self._detach_from_live_and_wait_for_event(

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2426, in
_detach_from_live_and_wait_for_event

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     self._detach_sync(

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2498, in
_detach_sync

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     guest.detach_device(dev,
persistent=persistent, live=live)

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
"/opt/stack/nova/nova/virt/libvirt/guest.py", line 441, in detach_device

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
self._domain.detachDeviceFlags(device_xml, flags=flags)

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
/dist-packages/eventlet/tpool.py", line 190, in doit

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     result =
proxy_call(self._autowrap, f, *args, **kwargs)

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
/dist-packages/eventlet/tpool.py", line 148, in proxy_call

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     rv = execute(f, *args,
**kwargs)

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
/dist-packages/eventlet/tpool.py", line 129, in execute

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR


nova.virt.block_device [instance: e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     six.reraise(c, e, tb)

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
/dist-packages/six.py", line 719, in reraise

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     raise value

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
/dist-packages/eventlet/tpool.py", line 83, in tworker

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     rv = meth(*args, **kwargs)

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
/dist-packages/libvirt.py", line 1515, in detachDeviceFlags

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     raise
libvirtError('virDomainDetachDeviceFlags() failed')

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5] libvirt.libvirtError: operation
failed: disk vdb not found

Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
compute[108182]: ERROR nova.virt.block_device [instance:
e3d6fd2b-7357-4d19-bcb4-482d97d332e5]

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure

** Tags added: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1931716

Title:
  Detaching volume from the live domain: virDomainDetachDeviceFlags():
  libvirt.libvirtError: operation failed: disk vdb not found

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  The test_attach_attached_volume_to_same_server tests within the nova-
  next job fail during detaching the volume:

  File "/usr/local/lib/python3.8/dist-packages/libvirt.py", line 1515,
  in detachDeviceFlags

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     raise
  libvirtError('virDomainDetachDeviceFlags() failed')

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5] libvirt.libvirtError: operation
  failed: disk vdb not found

  
  Steps to reproduce
  ==================

  Only seen as part of the nova-next job at present.

  Expected result
  ===============

  detach succeeds

  Actual result
  =============

  test fails as the volume state

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

  master

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

  libvirt

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

  cinder volume

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

  N/A

  Logs & Configs
  ==============

  https://zuul.opendev.org/t/openstack/build/02e6a99bf1574c978c663eb434705cbb/log/controller/logs/screen-n-cpu.txt?severity=0#34811

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [None req-aceedc3b-f03b-
  4bd5-8e8e-cfe66170a2e2 tempest-AttachVolumeNegativeTest-1273499975
  tempest-AttachVolumeNegativeTest-1273499975-project] [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5] Failed to detach volume
  2939aa92-0156-45d6-9689-ed48dcc8fd8a from /dev/vdb:
  libvirt.libvirtError: operation failed: disk vdb not found

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5] Traceback (most recent call
  last):

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/block_device.py", line 328, in
  driver_detach

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
  virt_driver.detach_volume(context, connection_info, instance, mp,

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2555, in
  detach_volume

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     self._detach_with_retry(

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2308, in
  _detach_with_retry

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
  self._detach_from_live_with_retry(

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2364, in
  _detach_from_live_with_retry

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
  self._detach_from_live_and_wait_for_event(

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2426, in
  _detach_from_live_and_wait_for_event

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     self._detach_sync(

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2498, in
  _detach_sync

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     guest.detach_device(dev,
  persistent=persistent, live=live)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/guest.py", line 441, in
  detach_device

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
  self._domain.detachDeviceFlags(device_xml, flags=flags)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 190, in doit

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     result =
  proxy_call(self._autowrap, f, *args, **kwargs)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 148, in proxy_call

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     rv = execute(f, *args,
  **kwargs)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 129, in execute

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR

  
  nova.virt.block_device [instance: e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     six.reraise(c, e, tb)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/six.py", line 719, in reraise

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     raise value

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 83, in tworker

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     rv = meth(*args, **kwargs)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/libvirt.py", line 1515, in detachDeviceFlags

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     raise
  libvirtError('virDomainDetachDeviceFlags() failed')

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5] libvirt.libvirtError: operation
  failed: disk vdb not found

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1931716/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next