yahoo-eng-team team mailing list archive

[OpenStack Infra <1931716@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/796255
Committed: https://opendev.org/openstack/nova/commit/eb67ec24e31d79749f2fa515fa69d861dfb12244
Submitter: "Zuul (22348)"
Branch:    master

commit eb67ec24e31d79749f2fa515fa69d861dfb12244
Author: Balazs Gibizer <balazs.gibizer@xxxxxxxx>
Date:   Mon Jun 14 12:29:18 2021 +0200

    Handle OPERATION_FAILED error during detach
    
    While libvirt 4.1 started to use DEVICE_MISSING for most of the cases
    there is still one[1] using OPERATION_FAILED when the device is missing
    during detach. So nova needs to handle that properly.
    
    Revert this when libvirt merges[1] and nova bumps the min version over
    to that release.
    
    [1] https://listman.redhat.com/archives/libvir-list/2021-June/msg00337.html
    
    Change-Id: Ibc967be291853e23027d9b8210ece572adcc80c9
    Closes-Bug: #1931716


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1931716

Title:
  Detaching volume from the live domain: virDomainDetachDeviceFlags():
  libvirt.libvirtError: operation failed: disk vdb not found

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========

  The test_attach_attached_volume_to_same_server tests within the nova-
  next job fail during detaching the volume:

  File "/usr/local/lib/python3.8/dist-packages/libvirt.py", line 1515,
  in detachDeviceFlags

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     raise
  libvirtError('virDomainDetachDeviceFlags() failed')

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5] libvirt.libvirtError: operation
  failed: disk vdb not found

  
  Steps to reproduce
  ==================

  Only seen as part of the nova-next job at present.

  Expected result
  ===============

  detach succeeds

  Actual result
  =============

  test fails as the volume state

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

  master

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

  libvirt

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

  cinder volume

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

  N/A

  Logs & Configs
  ==============

  https://zuul.opendev.org/t/openstack/build/02e6a99bf1574c978c663eb434705cbb/log/controller/logs/screen-n-cpu.txt?severity=0#34811

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [None req-aceedc3b-f03b-
  4bd5-8e8e-cfe66170a2e2 tempest-AttachVolumeNegativeTest-1273499975
  tempest-AttachVolumeNegativeTest-1273499975-project] [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5] Failed to detach volume
  2939aa92-0156-45d6-9689-ed48dcc8fd8a from /dev/vdb:
  libvirt.libvirtError: operation failed: disk vdb not found

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5] Traceback (most recent call
  last):

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/block_device.py", line 328, in
  driver_detach

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
  virt_driver.detach_volume(context, connection_info, instance, mp,

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2555, in
  detach_volume

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     self._detach_with_retry(

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2308, in
  _detach_with_retry

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
  self._detach_from_live_with_retry(

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2364, in
  _detach_from_live_with_retry

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
  self._detach_from_live_and_wait_for_event(

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2426, in
  _detach_from_live_and_wait_for_event

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     self._detach_sync(

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2498, in
  _detach_sync

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     guest.detach_device(dev,
  persistent=persistent, live=live)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File
  "/opt/stack/nova/nova/virt/libvirt/guest.py", line 441, in
  detach_device

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]
  self._domain.detachDeviceFlags(device_xml, flags=flags)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 190, in doit

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     result =
  proxy_call(self._autowrap, f, *args, **kwargs)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 148, in proxy_call

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     rv = execute(f, *args,
  **kwargs)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 129, in execute

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR

  
  nova.virt.block_device [instance: e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     six.reraise(c, e, tb)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/six.py", line 719, in reraise

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     raise value

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/eventlet/tpool.py", line 83, in tworker

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     rv = meth(*args, **kwargs)

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]   File "/usr/local/lib/python3.8
  /dist-packages/libvirt.py", line 1515, in detachDeviceFlags

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]     raise
  libvirtError('virDomainDetachDeviceFlags() failed')

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5] libvirt.libvirtError: operation
  failed: disk vdb not found

  Jun 11 13:30:47.071650 ubuntu-focal-inap-mtl01-0025074130 nova-
  compute[108182]: ERROR nova.virt.block_device [instance:
  e3d6fd2b-7357-4d19-bcb4-482d97d332e5]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1931716/+subscriptions