yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1673090] [NEW] Swap disk on stopped instance fails silently

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Steve Noyes <steve.noyes@xxxxxxxxxx>
Date: Wed, 15 Mar 2017 14:28:47 -0000
Reply-to: Bug 1673090 <1673090@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Using either the cli or API, if you attempt to swap disks (nova volume-
update, compute.api.swap_volume) on a stopped instance, the command
appears to succeed, but the swap is not performed. The way you can tell
what didn't happen is that the volume status's of the 2 volumes remain
as they were before the operation.

It would be better to throw an exception when a swap is attempted on a
stopped instance. (this can get tricky since an instance can stop at any
time during the swap operation. btw, it seems odd that attach and detach
are supported on a stopped instance but swap is not. would it be better
to support swap on a stopped instance?)

In the compute log is this exception:

2017-03-15 09:20:13.729 ERROR nova.compute.manager [^[[01;36mreq-e978c2af-d537-4855-ba5f-58ec1976ad2f ^[[00;36mtempest-SwapVolumeTestJSON-1034316961 tempest-SwapVolumeTestJSON-1034316961] ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] Failed to swap volume 11cda3f1-15c3-4195-b404-a7e912f4c517 for 8b6a1904-3e54-43f4-b498-0fa153d7cf17^[[00m
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00mTraceback (most recent call last):
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 4988, in _swap_volume
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    resize_to)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1300, in swap_volume
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    self._swap_volume(guest, disk_dev, conf.source_path, resize_to)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1257, in _swap_volume
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    dev.rebase(new_path, copy=True, reuse_ext=True)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 748, in rebase
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    self._disk, base, self.REBASE_DEFAULT_BANDWIDTH, flags=flags)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    rv = execute(f, *args, **kwargs)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    six.reraise(c, e, tb)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    rv = meth(*args, **kwargs)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 930, in blockRebase
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00mlibvirtError: Domain not found: no domain with matching uuid '26fe837e-d72c-4969-8934-09dc92b61897' (instance-00000022)
2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897]

My setup is devstack, libvirt+kvm, the volume is a cinder iscsi volume,
instance is cirros,

nova$ git log -1
commit a3655c311454c09312f12f413535d83302e1fba4
Merge: 1ef9cbd 3c7b73b
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date:   Tue Feb 28 19:21:35 2017 +0000

A somewhat related bug:

https://bugs.launchpad.net/nova/+bug/1635657

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1673090

Title:
  Swap disk on stopped instance fails silently

Status in OpenStack Compute (nova):
  New

Bug description:
  Using either the cli or API, if you attempt to swap disks (nova
  volume-update, compute.api.swap_volume) on a stopped instance, the
  command appears to succeed, but the swap is not performed. The way you
  can tell what didn't happen is that the volume status's of the 2
  volumes remain as they were before the operation.

  It would be better to throw an exception when a swap is attempted on a
  stopped instance. (this can get tricky since an instance can stop at
  any time during the swap operation. btw, it seems odd that attach and
  detach are supported on a stopped instance but swap is not. would it
  be better to support swap on a stopped instance?)

  In the compute log is this exception:

  2017-03-15 09:20:13.729 ERROR nova.compute.manager [^[[01;36mreq-e978c2af-d537-4855-ba5f-58ec1976ad2f ^[[00;36mtempest-SwapVolumeTestJSON-1034316961 tempest-SwapVolumeTestJSON-1034316961] ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] Failed to swap volume 11cda3f1-15c3-4195-b404-a7e912f4c517 for 8b6a1904-3e54-43f4-b498-0fa153d7cf17^[[00m
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00mTraceback (most recent call last):
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 4988, in _swap_volume
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    resize_to)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1300, in swap_volume
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    self._swap_volume(guest, disk_dev, conf.source_path, resize_to)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1257, in _swap_volume
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    dev.rebase(new_path, copy=True, reuse_ext=True)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 748, in rebase
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    self._disk, base, self.REBASE_DEFAULT_BANDWIDTH, flags=flags)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    rv = execute(f, *args, **kwargs)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    six.reraise(c, e, tb)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    rv = meth(*args, **kwargs)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m  File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 930, in blockRebase
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00m    if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897] ^[[00mlibvirtError: Domain not found: no domain with matching uuid '26fe837e-d72c-4969-8934-09dc92b61897' (instance-00000022)
  2017-03-15 09:20:13.729 TRACE nova.compute.manager ^[[01;35m[instance: 26fe837e-d72c-4969-8934-09dc92b61897]

  My setup is devstack, libvirt+kvm, the volume is a cinder iscsi
  volume, instance is cirros,

  nova$ git log -1
  commit a3655c311454c09312f12f413535d83302e1fba4
  Merge: 1ef9cbd 3c7b73b
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date:   Tue Feb 28 19:21:35 2017 +0000

  A somewhat related bug:

  https://bugs.launchpad.net/nova/+bug/1635657

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1673090/+subscriptions
Follow ups

[Bug 1673090] Re: Swap disk on stopped instance fails silently
From: OpenStack Infra, 2018-01-15