← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #77439

[Bug 1819569] [NEW] Swap volumes of vm failed: pivot of disk 'vdb' requires an active copy job

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========

I had one volume, vol-1 attached to two VMs, vm-1 and vm-2, then tried
to swap vm-1's volume to another volume vol-2, but failed.

The Cinder backend attached the volume to vm successfully.

Nova reported below error:

Mar 08 01:34:42.984286 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver [None req-1033d1c1-bd3b-4990-a8da-e503d25a041b tempest-TestMultiAttachVolumeSwap-890516006 tempest-TestMultiAttachVolumeSwap-890516006] Failure rebasing volume /dev/sdj on vdb.: libvirt.libvirtError: Requested operation is not valid: pivot of disk 'vdb' requires an active copy job
Mar 08 01:34:42.984521 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
Mar 08 01:34:42.984846 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1557, in _swap_volume
Mar 08 01:34:42.985032 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     dev.abort_job(pivot=True)
Mar 08 01:34:42.985214 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 754, in abort_job
Mar 08 01:34:42.986537 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     self._guest._domain.blockJobAbort(self._disk, flags=flags)
Mar 08 01:34:42.986914 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 190, in doit
Mar 08 01:34:42.987246 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     result = proxy_call(self._autowrap, f, *args, **kwargs)
Mar 08 01:34:42.987608 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 148, in proxy_call
Mar 08 01:34:42.987958 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     rv = execute(f, *args, **kwargs)
Mar 08 01:34:42.988276 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 129, in execute
Mar 08 01:34:42.988434 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     six.reraise(c, e, tb)
Mar 08 01:34:42.988591 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
Mar 08 01:34:42.988746 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     raise value
Mar 08 01:34:42.988895 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 83, in tworker
Mar 08 01:34:42.989045 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     rv = meth(*args, **kwargs)
Mar 08 01:34:42.989194 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/libvirt.py", line 772, in blockJobAbort
Mar 08 01:34:42.989366 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)
Mar 08 01:34:42.989516 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver libvirt.libvirtError: Requested operation is not valid: pivot of disk 'vdb' requires an active copy job


Steps to reproduce
==================

1. In OpenStack, attach a volume, vol-1 to two vms, vm-1 and vm-2. Volume vol-1 is provisioned on external storage system.
2. Swap vol-1 of vm-1 to another volume, vol-2.


Actual results
==============

You can see above failure: Failure rebasing volume /dev/sdj (vol-2) on
vdb.


Expected results
================
vm-1's volume updated to vol-2.


Environment
===========

OpenStack: master (Stein) devstack
libvirt-python: 5.1.0
libvirt-bin: 4.0.0-1ubuntu8.5~cloud0


Logs
====

1. Full n-cpu log locates: http://publiclogs.emc.com/51/633451/5/check/EMC_Unity_iSCSI/a2d0e05/EMC_Unity_iSCSI/1218/logs/screen-n-cpu.txt.gz
Search via request id: req-1033d1c1-bd3b-4990-a8da-e503d25a041b

2. libvirt log is attached.

If you need more info, please let me know.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: libvirt

** Attachment added: "libvirtd log"
   https://bugs.launchpad.net/bugs/1819569/+attachment/5245583/+files/libvirtd.log

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1819569

Title:
  Swap volumes of vm failed: pivot of disk 'vdb' requires an active copy
  job

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  I had one volume, vol-1 attached to two VMs, vm-1 and vm-2, then tried
  to swap vm-1's volume to another volume vol-2, but failed.

  The Cinder backend attached the volume to vm successfully.

  Nova reported below error:

  Mar 08 01:34:42.984286 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver [None req-1033d1c1-bd3b-4990-a8da-e503d25a041b tempest-TestMultiAttachVolumeSwap-890516006 tempest-TestMultiAttachVolumeSwap-890516006] Failure rebasing volume /dev/sdj on vdb.: libvirt.libvirtError: Requested operation is not valid: pivot of disk 'vdb' requires an active copy job
  Mar 08 01:34:42.984521 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver Traceback (most recent call last):
  Mar 08 01:34:42.984846 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 1557, in _swap_volume
  Mar 08 01:34:42.985032 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     dev.abort_job(pivot=True)
  Mar 08 01:34:42.985214 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 754, in abort_job
  Mar 08 01:34:42.986537 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     self._guest._domain.blockJobAbort(self._disk, flags=flags)
  Mar 08 01:34:42.986914 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 190, in doit
  Mar 08 01:34:42.987246 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Mar 08 01:34:42.987608 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 148, in proxy_call
  Mar 08 01:34:42.987958 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     rv = execute(f, *args, **kwargs)
  Mar 08 01:34:42.988276 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 129, in execute
  Mar 08 01:34:42.988434 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     six.reraise(c, e, tb)
  Mar 08 01:34:42.988591 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
  Mar 08 01:34:42.988746 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     raise value
  Mar 08 01:34:42.988895 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/eventlet/tpool.py", line 83, in tworker
  Mar 08 01:34:42.989045 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     rv = meth(*args, **kwargs)
  Mar 08 01:34:42.989194 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver   File "/usr/local/lib/python3.5/dist-packages/libvirt.py", line 772, in blockJobAbort
  Mar 08 01:34:42.989366 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver     if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self)
  Mar 08 01:34:42.989516 UnityiSCSISlave nova-compute[19437]: ERROR nova.virt.libvirt.driver libvirt.libvirtError: Requested operation is not valid: pivot of disk 'vdb' requires an active copy job

  
  Steps to reproduce
  ==================

  1. In OpenStack, attach a volume, vol-1 to two vms, vm-1 and vm-2. Volume vol-1 is provisioned on external storage system.
  2. Swap vol-1 of vm-1 to another volume, vol-2.

  
  Actual results
  ==============

  You can see above failure: Failure rebasing volume /dev/sdj (vol-2) on
  vdb.

  
  Expected results
  ================
  vm-1's volume updated to vol-2.

  
  Environment
  ===========

  OpenStack: master (Stein) devstack
  libvirt-python: 5.1.0
  libvirt-bin: 4.0.0-1ubuntu8.5~cloud0

  
  Logs
  ====

  1. Full n-cpu log locates: http://publiclogs.emc.com/51/633451/5/check/EMC_Unity_iSCSI/a2d0e05/EMC_Unity_iSCSI/1218/logs/screen-n-cpu.txt.gz
  Search via request id: req-1033d1c1-bd3b-4990-a8da-e503d25a041b

  2. libvirt log is attached.

  If you need more info, please let me know.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1819569/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next