yahoo-eng-team team mailing list archive

[Pavlo Shchelokovskyy <2120167@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

We started seeing this in downstream specifically starting with Epoxy
and only for volumes mounted through iscsi (LVM cinder backend in our
case), RBD works fine. It is reliably reproducible on current
devstack/master running on Ubuntu Noble too.

Also the error goes away if we downgrade libvirt from 10.x (packaged by
Ubuntu Noble) to 8.x (packaged by Ubuntu Jammy), so this might be smth
specific to new(er) libvirt.

In particular, the test

tempest.api.compute.admin.test_volume_swap.TestVolumeSwap.test_volume_swap

reliably fails on the second volume swap for iscsi-mounted volumes with
the following trace:

   raise libvirtError('virDomainBlockCopy() failed')
 libvirt.libvirtError: Requested operation is not valid: Setting different DAC user or group on /dev/sda which is already in use

Setting remember_owner = 0 in /etc/libvirt/qemu.conf fixes the problem
too, but I am not sure if this is safe enough and if it could break
other operations that require nova access to a disk of a shutdown/paused
vm.

On a second thought, the tempest test itself seems rather synthetic.
This volume swap API is not meant for end users, but only as part of
orchestrating a volume migration/retype etc. And across all openstack
I've found only two places that call this API - cinder and watcher, and
neither of those makes this swap back.

Indeed, if you think of it, swapping back means that you first rebase
volume1 on volume2, and then you try to rebase volume2 on volume1,
effectively trying to rebase the volume1 on itself, which does not sound
right...

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2120167

Title:
  blockCopy fails during second volume swap in test_volume_swap Tempest
  test

Status in OpenStack Compute (nova):
  New

Bug description:
  We started seeing this in downstream specifically starting with Epoxy
  and only for volumes mounted through iscsi (LVM cinder backend in our
  case), RBD works fine. It is reliably reproducible on current
  devstack/master running on Ubuntu Noble too.

  Also the error goes away if we downgrade libvirt from 10.x (packaged
  by Ubuntu Noble) to 8.x (packaged by Ubuntu Jammy), so this might be
  smth specific to new(er) libvirt.

  In particular, the test

  tempest.api.compute.admin.test_volume_swap.TestVolumeSwap.test_volume_swap

  reliably fails on the second volume swap for iscsi-mounted volumes
  with the following trace:

     raise libvirtError('virDomainBlockCopy() failed')
   libvirt.libvirtError: Requested operation is not valid: Setting different DAC user or group on /dev/sda which is already in use

  Setting remember_owner = 0 in /etc/libvirt/qemu.conf fixes the problem
  too, but I am not sure if this is safe enough and if it could break
  other operations that require nova access to a disk of a
  shutdown/paused vm.

  On a second thought, the tempest test itself seems rather synthetic.
  This volume swap API is not meant for end users, but only as part of
  orchestrating a volume migration/retype etc. And across all openstack
  I've found only two places that call this API - cinder and watcher,
  and neither of those makes this swap back.

  Indeed, if you think of it, swapping back means that you first rebase
  volume1 on volume2, and then you try to rebase volume2 on volume1,
  effectively trying to rebase the volume1 on itself, which does not
  sound right...

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2120167/+subscriptions