← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #80564

[Bug 1850783] [NEW] Need to rollback persistent domain xml after DeviceDetachFailed exception

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
Volume status rollbacks to "in-use" after a failure to detach a volume due to DeviceDetachFailed, which is fixed in https://bugs.launchpad.net/nova/+bug/1786318.
But the next detaching action will fail because of DeviceNotFound exception.

Steps to reproduce
==================
Attempt to detach a volume while it is mounted and in-use within an instance. 
Wait volume status rollbacks to "in-use".
Attempt to detach the volume again.


Expected result
===============
Volume detach succeeds.

Actual result
=============
Volume detach fails and the volume returns to in-use after the second detach.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   Queens-17.0.3

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   N/A

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============

Logs of second detach after DeviceDetachFailed

2019-10-30 13:17:29.401 2109249 INFO nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Attempting to driver detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from mountpoint /dev/vdb
2019-10-30 13:17:29.407 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] Attempting initial detach for device vdb detach_device_with_retry /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:435
2019-10-30 13:17:29.408 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] detach device xml: <disk type="network" device="disk">
  <driver name="qemu" type="raw" cache="writethrough" discard="unmap"/>
  <source protocol="rbd" name="volumes/volume-355d2619-cd59-4c97-80a9-b65ff1870961">
    <host name="10.114.198.109" port="6789"/>
    <host name="10.114.198.110" port="6789"/>
    <host name="10.114.198.111" port="6789"/>
    <host name="10.114.198.116" port="6789"/>
    <host name="10.114.198.117" port="6789"/>
  </source>
  <target bus="virtio" dev="vdb"/>
  <serial>355d2619-cd59-4c97-80a9-b65ff1870961</serial>
  <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
</disk>
 detach_device /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:486
2019-10-30 13:17:29.417 2109249 ERROR nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Failed to detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from /dev/vdb: libvirtError: device not found: no target device vdb

** Affects: nova
     Importance: Undecided
     Assignee: Fan Zhang (fanzhang)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Fan Zhang (fanzhang)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1850783

Title:
  Need to rollback persistent domain xml after DeviceDetachFailed
  exception

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  Volume status rollbacks to "in-use" after a failure to detach a volume due to DeviceDetachFailed, which is fixed in https://bugs.launchpad.net/nova/+bug/1786318.
  But the next detaching action will fail because of DeviceNotFound exception.

  Steps to reproduce
  ==================
  Attempt to detach a volume while it is mounted and in-use within an instance. 
  Wait volume status rollbacks to "in-use".
  Attempt to detach the volume again.

  
  Expected result
  ===============
  Volume detach succeeds.

  Actual result
  =============
  Volume detach fails and the volume returns to in-use after the second detach.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     Queens-17.0.3

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libirt + KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     N/A

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============

  Logs of second detach after DeviceDetachFailed

  2019-10-30 13:17:29.401 2109249 INFO nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Attempting to driver detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from mountpoint /dev/vdb
  2019-10-30 13:17:29.407 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] Attempting initial detach for device vdb detach_device_with_retry /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:435
  2019-10-30 13:17:29.408 2109249 DEBUG nova.virt.libvirt.guest [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] detach device xml: <disk type="network" device="disk">
    <driver name="qemu" type="raw" cache="writethrough" discard="unmap"/>
    <source protocol="rbd" name="volumes/volume-355d2619-cd59-4c97-80a9-b65ff1870961">
      <host name="10.114.198.109" port="6789"/>
      <host name="10.114.198.110" port="6789"/>
      <host name="10.114.198.111" port="6789"/>
      <host name="10.114.198.116" port="6789"/>
      <host name="10.114.198.117" port="6789"/>
    </source>
    <target bus="virtio" dev="vdb"/>
    <serial>355d2619-cd59-4c97-80a9-b65ff1870961</serial>
    <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
  </disk>
   detach_device /usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py:486
  2019-10-30 13:17:29.417 2109249 ERROR nova.virt.block_device [None req-a171d8e0-c544-4769-a31c-e1d9c30afad9 6bc9c2831be24f7586bc3ce1dd38b375 d7c7d7f7d3b044c0a1ee7833cfb46f32 - default default] [instance: 873dc3b8-7d58-4410-9c07-88f8dd2779dc] Failed to detach volume 355d2619-cd59-4c97-80a9-b65ff1870961 from /dev/vdb: libvirtError: device not found: no target device vdb

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1850783/+subscriptions
  Thread Previous • Date Previous • • Thread Next