yahoo-eng-team team mailing list archive

[Matt Riedemann <mriedem@xxxxxxxxxx>]

Nova has the same issue with local lvm block storage:

http://logs.openstack.org/80/237480/8/check/gate-tempest-dsvm-
full/96d1bff/logs/screen-n-cpu.txt.gz?level=TRACE

** Also affects: nova
   Importance: Undecided
       Status: New

** Tags added: volumes

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1270192

Title:
  cinder volume hanging on removing snapshots

Status in Cinder:
  Fix Released
Status in Cinder icehouse series:
  Fix Released
Status in OpenStack Compute (nova):
  New

Bug description:
  On any lvm2 version without lvmetad running, I can get cinder-volume
  to hang on issuing lvm related commands after deleting snapshots (that
  are non-thin provisioned LVM snapshots with clear_volume set to zero).

  the issue is that lvm locks up due to trying to access suspended device mapper entries, and at some point cinder-volume does a lvm related command and hangs on that. a setting of ignore_suspended_devices = 1 in lvm.conf helps with that, as lvremove hangs on scanning the device state (which it
  needs to do because it doesn't have current information available via lvmetad).

  I can use this script to trigger the issue:

  === cut hang.sh ===
  enable_fix=1
  #enable_fix=0

  vg=cinder-volumes
  v=testvol.$$

  lvcreate --name $v $vg -L 1g
  sleep 2
  lvcreate --name snap-$v --snapshot $vg/$v -L 1g

  vgp=/dev/mapper/${vg/-/--}-snap--${v/-/--}

  sleep 2

  ( sleep 10 < $vgp-cow ) &
  test "$enable_fix" -eq "1" && lvchange -y -an $vg/snap-$v
  lvremove -f $vg/snap-$v
  sleep 1
  lvremove -f $vg/$v
  === cut hang.sh ===

  vg needs to be set to a lvm VG that exists and can take a few gig of
  space. whenever enable_fix is set to 0, lvremove -f ends with :

    Unable to deactivate open cinder--volumes-snap--testvol.27700-cow (252:5)
    Failed to resume snap-testvol.27700.
    libdevmapper exiting with 1 device(s) still suspended.

  this is because the sleep command before keeps a fd open on the -cow.
  The script then also never finishes and any other lvm command hangs as
  well.

  apparently in real-life this is either udev or the dd command still
  having the fd open for some reason I have not yet understood.

  The deactivation before removing seems to help.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1270192/+subscriptions