yahoo-eng-team team mailing list archive

[Stephen Finucane <2119114@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

As discussed in detail on a k8s cloud-provider-openstack bug [1], it
appears that Nova always times out when waiting for libvirt events
during a disk detach, with the following warning appearing in logs:


```
Jul 24 15:48:56 stephenfin-cpo-debug-new nova-compute[68061]: INFO nova.virt.block_device [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] [instance: e8d15440-1442-49c3-9356-4592cdb697d2] Attempting to driver detach volume df0017a1-b3c5-4f30-b48c-7c6da9c763a2 from mountpoint /dev/vdb
Jul 24 15:49:16 stephenfin-cpo-debug-new nova-compute[68061]: WARNING nova.virt.libvirt.driver [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] Waiting for libvirt event about the detach of device vdb with device alias ua-df0017a1-b3c5-4f30-b48c-7c6da9c763a2 from instance e8d15440-1442-49c3-9356-4592cdb697d2 is timed out.
Jul 24 15:49:16 stephenfin-cpo-debug-new nova-compute[68061]: INFO nova.virt.libvirt.driver [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] Successfully detached device vdb from instance e8d15440-1442-49c3-9356-4592cdb697d2 from the live domain config.
```


This appears to have started in 2024.1 (Caracal) and appears regardless of distro. The net of effect of this is a 20 second (by default) timeout for each disk detach. This is proving particularly problematic for CPO since our tests do a lot of attach/detach calls (for creating/deleting PVCs) and run tests in series, however, we also see the same issue in the Nova CI.

[1] https://github.com/kubernetes/cloud-provider-
openstack/pull/2742#issuecomment-3135940269

** Affects: nova
     Importance: Undecided
         Status: Confirmed


** Tags: libvirt

** Changed in: nova
       Status: New => Confirmed

** Tags added: libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2119114

Title:
  Consistent timeouts waiting for libvirt event during disk detach

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  As discussed in detail on a k8s cloud-provider-openstack bug [1], it
  appears that Nova always times out when waiting for libvirt events
  during a disk detach, with the following warning appearing in logs:

  
  ```
  Jul 24 15:48:56 stephenfin-cpo-debug-new nova-compute[68061]: INFO nova.virt.block_device [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] [instance: e8d15440-1442-49c3-9356-4592cdb697d2] Attempting to driver detach volume df0017a1-b3c5-4f30-b48c-7c6da9c763a2 from mountpoint /dev/vdb
  Jul 24 15:49:16 stephenfin-cpo-debug-new nova-compute[68061]: WARNING nova.virt.libvirt.driver [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] Waiting for libvirt event about the detach of device vdb with device alias ua-df0017a1-b3c5-4f30-b48c-7c6da9c763a2 from instance e8d15440-1442-49c3-9356-4592cdb697d2 is timed out.
  Jul 24 15:49:16 stephenfin-cpo-debug-new nova-compute[68061]: INFO nova.virt.libvirt.driver [None req-672548ab-5a3e-4698-b5cd-0127af0d4358 demo demo] Successfully detached device vdb from instance e8d15440-1442-49c3-9356-4592cdb697d2 from the live domain config.
  ```

  
  This appears to have started in 2024.1 (Caracal) and appears regardless of distro. The net of effect of this is a 20 second (by default) timeout for each disk detach. This is proving particularly problematic for CPO since our tests do a lot of attach/detach calls (for creating/deleting PVCs) and run tests in series, however, we also see the same issue in the Nova CI.

  [1] https://github.com/kubernetes/cloud-provider-
  openstack/pull/2742#issuecomment-3135940269

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2119114/+subscriptions