yahoo-eng-team team mailing list archive

[Lee Yarwood <lyarwood@xxxxxxxxxx>]

Public bug reported:

Description
===========
The following was noticed during a CI run for https://review.openstack.org/#/c/383859/ :

http://logs.openstack.org/09/395709/7/check/gate-tempest-dsvm-full-
devstack-plugin-nfs-
nv/a4c1057/logs/screen-n-cpu.txt.gz?level=ERROR#_2017-02-07_19_17_41_994

This is due to rebuild calling for two separate detach/disconnects of a
volume when using the libvirt virt driver, once in _rebuild_default_impl
in the compute layer and a second time in cleanup within the virt driver
:

https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2653 - _rebuild_default_impl
https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L989 - cleanup

In the logs req-e976fee4-51df-4119-b505-5d68f4583186 tracks the rebuild
attempt. We see the first attempt to umount succeed here :

http://logs.openstack.org/09/395709/7/check/gate-tempest-dsvm-full-
devstack-plugin-nfs-
nv/a4c1057/logs/screen-n-cpu.txt.gz?level=ALL#_2017-02-07_19_17_39_904

We then see the second attempt here and again an ERROR is logged as we
don't find the mount to be in use :

http://logs.openstack.org/09/395709/7/check/gate-tempest-dsvm-full-
devstack-plugin-nfs-
nv/a4c1057/logs/screen-n-cpu.txt.gz?level=ALL#_2017-02-07_19_17_41_993

Steps to reproduce
==================
Rebuild an instance with volumes attached

Expected result
===============
Only one attempt is made to detach and disconnect each volume from the original instance.

Actual result
=============
Two attempts are made to detach and disconnect each volume from the original instance.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   https://review.openstack.org/#/c/383859/ - but it should reproduce
against master.

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   n/a

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   n/a

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1662869

Title:
  Multiple attempts to detach and disconnect volumes during rebuild

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  The following was noticed during a CI run for https://review.openstack.org/#/c/383859/ :

  http://logs.openstack.org/09/395709/7/check/gate-tempest-dsvm-full-
  devstack-plugin-nfs-
  nv/a4c1057/logs/screen-n-cpu.txt.gz?level=ERROR#_2017-02-07_19_17_41_994

  This is due to rebuild calling for two separate detach/disconnects of
  a volume when using the libvirt virt driver, once in
  _rebuild_default_impl in the compute layer and a second time in
  cleanup within the virt driver :

  https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2653 - _rebuild_default_impl
  https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L989 - cleanup

  In the logs req-e976fee4-51df-4119-b505-5d68f4583186 tracks the
  rebuild attempt. We see the first attempt to umount succeed here :

  http://logs.openstack.org/09/395709/7/check/gate-tempest-dsvm-full-
  devstack-plugin-nfs-
  nv/a4c1057/logs/screen-n-cpu.txt.gz?level=ALL#_2017-02-07_19_17_39_904

  We then see the second attempt here and again an ERROR is logged as we
  don't find the mount to be in use :

  http://logs.openstack.org/09/395709/7/check/gate-tempest-dsvm-full-
  devstack-plugin-nfs-
  nv/a4c1057/logs/screen-n-cpu.txt.gz?level=ALL#_2017-02-07_19_17_41_993

  Steps to reproduce
  ==================
  Rebuild an instance with volumes attached

  Expected result
  ===============
  Only one attempt is made to detach and disconnect each volume from the original instance.

  Actual result
  =============
  Two attempts are made to detach and disconnect each volume from the original instance.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     https://review.openstack.org/#/c/383859/ - but it should reproduce
  against master.

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     n/a

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     n/a

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1662869/+subscriptions