yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1661016] Re: Multiple attempts to swap volumes using volume-update fail

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Fri, 10 Feb 2017 14:42:31 -0000
Reply-to: Bug 1661016 <1661016@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: nova/ocata
   Importance: Undecided
       Status: New

** Also affects: nova/newton
   Importance: Undecided
       Status: New

** Tags added: libvirt swap-volume volumes

** Changed in: nova/newton
       Status: New => Confirmed

** Changed in: nova/ocata
       Status: New => In Progress

** Changed in: nova/newton
     Assignee: (unassigned) => Lee Yarwood (lyarwood)

** Changed in: nova/newton
       Status: Confirmed => In Progress

** Changed in: nova/ocata
     Assignee: (unassigned) => Lee Yarwood (lyarwood)

** Changed in: nova
   Importance: Undecided => Medium

** Changed in: nova/ocata
   Importance: Undecided => Medium

** Changed in: nova/newton
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1661016

Title:
  Multiple attempts to swap volumes using volume-update fail

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) newton series:
  In Progress
Status in OpenStack Compute (nova) ocata series:
  In Progress

Bug description:
  Description
  ===========
  The second and any future attempts to swap volumes using volume-update fail due to a BDM lookup failure using the original volume id (see Logs & Configs for an example).

  A previous attempt to fix this was made in bug#1490236 and reverted by
  bug#1625660.

  Steps to reproduce
  ==================
  - Boot an instance
  - Create multiple volumes
  - Attach a single volume
  - Swap the attached volume with one that is unattached via nova volume-update.
  - Swap the attached volume with one that is unattached via nova volume-update.

  Expected result
  ===============
  The second attempt succeeds and the new volume is now attached to the instance.

  Actual result
  =============
  The second attempt fails looking up a BDM with the ID of the original volume.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/
     $ pwd
     /opt/stack/nova
     $ git rev-parse HEAD
     dae6b760b9c40bbf3b72a0218dbf1dbc823f30e2

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt + KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     LVM/iSCSI

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     n/a

  Logs & Configs
  ==============
  $ nova boot --image cirros-0.3.4-x86_64-uec --flavor 1 test-boot
  $ cinder create 1 ; cinder create 1

  $ nova volume-attach ef426f1e-32e4-4f8c-a3fc-b58080d38294 \
                       23933e67-a4c0-4de9-b0dc-3da37bce1b78

  $ nova volume-update ef426f1e-32e4-4f8c-a3fc-b58080d38294 \
                       23933e67-a4c0-4de9-b0dc-3da37bce1b78 \
                       cace165f-9c97-4d6d-a0e8-ea087fa80263

  $ nova volume-update ef426f1e-32e4-4f8c-a3fc-b58080d38294 \
                       cace165f-9c97-4d6d-a0e8-ea087fa80263 \ 
                       23933e67-a4c0-4de9-b0dc-3da37bce1b78

  n-cpu.log :

  4448 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
  4449 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 155, in _process_incoming
  4450 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  4451 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 222, in dispatch
  4452 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  4453 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 192, in _do_dispatch
  4454 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  4455 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 75, in wrapped
  4456 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     function_name, call_dict, binary)
  4457 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  4458 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     self.force_reraise()
  4459 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  4460 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  4461 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/exception_wrapper.py", line 66, in wrapped
  4462 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  4463 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 188, in decorated_function
  4464 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     LOG.warning(msg, e, instance=instance)
  4465 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  4466 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     self.force_reraise()
  4467 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  4468 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  4469 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 157, in decorated_function
  4470 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  4471 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 216, in decorated_function
  4472 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  4473 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  4474 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     self.force_reraise()
  4475 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  4476 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  4477 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 204, in decorated_function
  4478 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  4479 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 5052, in swap_volume
  4480 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     resize_to)          
  4481 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4998, in _swap_volume
  4482 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     self.volume_api.unreserve_volume(context, new_volume_id)
  4483 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  4484 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     self.force_reraise()
  4485 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  4486 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  4487 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/compute/manager.py", line 4976, in _swap_volume
  4488 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     resize_to)          
  4489 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1296, in swap_volume
  4490 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     nova_context.get_admin_context(), volume_id, instance.uuid)
  4491 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  4492 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     args, kwargs)       
  4493 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/conductor/rpcapi.py", line 239, in object_class_action_versions
  4494 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     args=args, kwargs=kwargs)
  4495 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 169, in call
  4496 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     retry=self.retry)   
  4497 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 97, in _send
  4498 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     timeout=timeout, retry=retry)
  4499 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in send
  4500 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     retry=retry)        
  4501 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 449, in _send
  4502 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     raise result        
  4503 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server VolumeBDMNotFound_Remote: No volume Block Device Mapping with id 23933e67-a4c0-4de9-b0dc-3da37bce1b78.
  4504 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
  4505 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server                         
  4506 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/conductor/manager.py", line 92, in _object_dispatch
  4507 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     return getattr(target, method)(*args, **kwargs)
  4508 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server                         
  4509 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  4510 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     result = fn(cls, context, *args, **kwargs)
  4511 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server                         
  4512 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server   File "/opt/stack/nova/nova/objects/block_device.py", line 242, in get_by_volume_and_instance
  4513 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server     raise exception.VolumeBDMNotFound(volume_id=volume_id)
  4514 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server                         
  4515 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server VolumeBDMNotFound: No volume Block Device Mapping with id 23933e67-a4c0-4de9-b0dc-3da37bce1b78.
  4516 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server                         
  4517 2017-02-01 07:35:04.931 TRACE oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1661016/+subscriptions
References

[Bug 1661016] [NEW] Multiple attempts to swap volumes using volume-update fail
From: Lee Yarwood, 2017-02-01