yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1490236] Re: Swapping volume can't been swap again

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Sun, 29 Jan 2017 19:37:44 -0000
Reply-to: Bug 1490236 <1490236@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/mitaka
   Importance: Undecided
       Status: New

** No longer affects: nova/mitaka

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1490236

Title:
  Swapping volume can't been swap again

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  If we had two volume, one is attached by a instance, and the other is still available.
  Due to https://bugs.launchpad.net/nova/+bug/1489744, those volumes will stay in wrong status.
  It will swap successful after fix the issue.
  Volumes will be available and in-use status.
  But when I try to swap in-use volume with other available volume.
  Nova compute will throw following exception:

  2015-08-30 04:55:13.772 ERROR oslo_messaging.rpc.dispatcher [req-4d999362-7a13-4b43-8c6a-0d85f3b9aa5b admin admin] Exception during message handling: No volume Block Device Mapping with id cafc833a-8645-47db-b464-999142afa7be.
  Traceback (most recent call last):

    File "/opt/stack/nova/nova/conductor/manager.py", line 443, in _object_dispatch
      return getattr(target, method)(*args, **kwargs)

    File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 169, in wrapper
      result = fn(cls, context, *args, **kwargs)

    File "/opt/stack/nova/nova/objects/block_device.py", line 204, in get_by_volume_id
      raise exception.VolumeBDMNotFound(volume_id=volume_id)

  VolumeBDMNotFound: No volume Block Device Mapping with id
  cafc833a-8645-47db-b464-999142afa7be.

  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 72, in wrapped
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 345, in decorated_function
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance_uuid=instance_uuid)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 316, in decorated_function
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 373, in decorated_function
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 361, in decorated_function
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4706, in swap_volume
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     resize_to)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4658, in _swap_volume
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     self.volume_api.unreserve_volume(context, new_volume_id)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4639, in _swap_volume
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     resize_to)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1217, in swap_volume
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     nova_context.get_admin_context(), volume_id)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 167, in wrapper
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     args, kwargs)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 239, in object_class_action
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     objver=objver, args=args, kwargs=kwargs)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     retry=self.retry)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     timeout=timeout, retry=retry)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 431, in send
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     retry=retry)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 422, in _send
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     raise result
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher VolumeBDMNotFound_Remote: No volume Block Device Mapping with id cafc833a-8645-47db-b464-999142afa7be.
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher 
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/manager.py", line 443, in _object_dispatch
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     return getattr(target, method)(*args, **kwargs)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher 
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 169, in wrapper
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     result = fn(cls, context, *args, **kwargs)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher 
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/objects/block_device.py", line 204, in get_by_volume_id
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher     raise exception.VolumeBDMNotFound(volume_id=volume_id)
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher 
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher VolumeBDMNotFound: No volume Block Device Mapping with id cafc833a-8645-47db-b464-999142afa7be.
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher 
  2015-08-30 04:55:13.772 TRACE oslo_messaging.rpc.dispatcher 

  In first swap, nova will save serial with old volume id in block device mapping.
  Therefore nova will try to find block device mapping through old volume id at second swap.
  It had been change to new volume id, so nova will raise no bdm not found.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1490236/+subscriptions
References

[Bug 1490236] [NEW] Swapping volume can't been swap again
From: Chung Chih, Hung, 2015-08-30