yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1939108] [NEW] tempest.api.compute.volumes.test_attach_volume_negative.AttachVolumeNegativeTest fails due to detach failure during cleanup

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <1939108@xxxxxxxxxxxxxxxxxx>
Date: Fri, 06 Aug 2021 07:54:44 -0000
Reply-to: Bug 1939108 <1939108@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Creating a fresh bug for this as we haven't seen it in a while since the
libvirt event rewrite landed but it did just cause a gate failure...

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5d6/801990/6/gate/tempest-
integrated-compute/5d6a7ec/testr_results.html

13 Aug 05 16:01:08.964854 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe      st-AttachVolumeNegativeTest-1260221999-project] Waiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f is timed out.
47114 Aug 05 16:01:08.969793 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe      st-AttachVolumeNegativeTest-1260221999-project] Failed to detach device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Libvirt did not re      port any error but the device is still in the config. {{(pid=104299) _detach_from_live_with_retry /opt/stack/nova/nova/virt/libvirt/driver.py:2394}}
47115 Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe      st-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. De      vice is still attached to the guest.
47116 Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe      st-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Devi      ce detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached       to the guest.
47117 Aug 05 16:01:09.024848 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG oslo_concurrency.lockutils [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tem      pest-AttachVolumeNegativeTest-1260221999-project] Lock "ce4b444f-1587-43e9-b708-d2ae2c9cd59f" released by "nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume" :: held 201.180s {{(p      id=104299) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:367}}
47118 Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 temp      est-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device       alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.


$ logsearch storedsearch bug-xxx-detach
Running stored search:
  bug-xxx-detach:
    branches:
    - master
    files:
    - controller/logs/screen-n-cpu.txt
    job-groups:
    - nova-devstack
    limit: 100
    project: openstack/nova
    regex: from the live domain config. Device is still attached to the guest.
    result: FAILURE
[..]
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47115:Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47116:Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47118:Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47170:Aug 05 16:01:09.101298 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.

Builds with matching logs 1/100:
+----------------------------------+---------------------+-----------------------------------+----------+----------------------------+
| uuid                             | finished            | review                            | pipeline | job                        |
+----------------------------------+---------------------+-----------------------------------+----------+----------------------------+
| 5d6a7ec36f104f10a1f49fe44909aa23 | 2021-08-05T16:25:36 | https://review.opendev.org/801990 | gate     | tempest-integrated-compute |
+----------------------------------+---------------------+-----------------------------------+----------+----------------------------+

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: gate-failure

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1939108

Title:
  tempest.api.compute.volumes.test_attach_volume_negative.AttachVolumeNegativeTest
  fails due to detach failure during cleanup

Status in OpenStack Compute (nova):
  New

Bug description:
  Creating a fresh bug for this as we haven't seen it in a while since
  the libvirt event rewrite landed but it did just cause a gate
  failure...

  https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_5d6/801990/6/gate/tempest-
  integrated-compute/5d6a7ec/testr_results.html

  13 Aug 05 16:01:08.964854 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe      st-AttachVolumeNegativeTest-1260221999-project] Waiting for libvirt event about the detach of device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f is timed out.
  47114 Aug 05 16:01:08.969793 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe      st-AttachVolumeNegativeTest-1260221999-project] Failed to detach device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Libvirt did not re      port any error but the device is still in the config. {{(pid=104299) _detach_from_live_with_retry /opt/stack/nova/nova/virt/libvirt/driver.py:2394}}
  47115 Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe      st-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. De      vice is still attached to the guest.
  47116 Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempe      st-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Devi      ce detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached       to the guest.
  47117 Aug 05 16:01:09.024848 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: DEBUG oslo_concurrency.lockutils [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tem      pest-AttachVolumeNegativeTest-1260221999-project] Lock "ce4b444f-1587-43e9-b708-d2ae2c9cd59f" released by "nova.compute.manager.ComputeManager.detach_volume.<locals>.do_detach_volume" :: held 201.180s {{(p      id=104299) inner /usr/local/lib/python3.8/dist-packages/oslo_concurrency/lockutils.py:367}}
  47118 Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 temp      est-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device       alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.

  
  $ logsearch storedsearch bug-xxx-detach
  Running stored search:
    bug-xxx-detach:
      branches:
      - master
      files:
      - controller/logs/screen-n-cpu.txt
      job-groups:
      - nova-devstack
      limit: 100
      project: openstack/nova
      regex: from the live domain config. Device is still attached to the guest.
      result: FAILURE
  [..]
  5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47115:Aug 05 16:01:08.970117 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR nova.virt.libvirt.driver [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
  5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47116:Aug 05 16:01:08.971007 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: WARNING nova.virt.block_device [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] [instance: ce4b444f-1587-43e9-b708-d2ae2c9cd59f] Guest refused to detach volume ff8d0867-1e92-4f04-8e52-01bf05f52ac3: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
  5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47118:Aug 05 16:01:09.095134 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server [None req-54714613-ee7e-4c9f-8e60-8251a77a526e tempest-AttachVolumeNegativeTest-1260221999 tempest-AttachVolumeNegativeTest-1260221999-project] Exception during message handling: nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.
  5d6a7ec36f104f10a1f49fe44909aa23:.logsearch/5d6a7ec36f104f10a1f49fe44909aa23/controller/logs/screen-n-cpu.txt:47170:Aug 05 16:01:09.101298 ubuntu-focal-ovh-gra1-0025782813 nova-compute[104299]: ERROR oslo_messaging.rpc.server nova.exception.DeviceDetachFailed: Device detach failed for vdb: Run out of retry while detaching device vdb with device alias virtio-disk1 from instance ce4b444f-1587-43e9-b708-d2ae2c9cd59f from the live domain config. Device is still attached to the guest.

  Builds with matching logs 1/100:
  +----------------------------------+---------------------+-----------------------------------+----------+----------------------------+
  | uuid                             | finished            | review                            | pipeline | job                        |
  +----------------------------------+---------------------+-----------------------------------+----------+----------------------------+
  | 5d6a7ec36f104f10a1f49fe44909aa23 | 2021-08-05T16:25:36 | https://review.opendev.org/801990 | gate     | tempest-integrated-compute |
  +----------------------------------+---------------------+-----------------------------------+----------+----------------------------+

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1939108/+subscriptions
Follow ups

[Bug 1939108] Related fix merged to nova (master)
From: OpenStack Infra, 2021-08-21
[Bug 1939108] Re: Failure to detach volume during Tempest test cleanup due to APIC related kernel panic within the guest OS
From: Lee Yarwood, 2021-08-20