yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1685185] [NEW] disconnect_volume not called when rebase failures are encountered during swap_volume

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date: Fri, 21 Apr 2017 11:07:39 -0000
Reply-to: Bug 1685185 <1685185@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========

At present disconnect_volume is not called when rebase failures are
encountered during swap_volume. This results in the new volume remaining
connected to the compute host prior to terminate_connection then being
called. This can in turn lead to left over devices remaining on the
compute host for some volume backends such as LVM/iSCSI.

Steps to reproduce
==================

Downstream, the easiest way to reproduce this is via
https://bugzilla.redhat.com/show_bug.cgi?id=1401755 :

# sudo setenforce 1
# nova update-volume ${instance_uuid} \
                     ${attached_NFS_volume_id} \
                     ${unattached_iSCSI_volume_id}

Upstream, I've been unable to get devstack to even work correctly with
SELinux in enforcing mode so I've been unable to reproduce the rollback
this way.

Expected result
===============

New volume disconnected from compute host.

Actual result
=============

New volume remains connected to compute host.

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/

   N/A

2. Which hypervisor did you use?
   (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
   What's the version of that?

   Libvirt + KVM

2. Which storage type did you use?
   (For example: Ceph, LVM, GPFS, ...)
   What's the version of that?

   NFS + LVM/iSCSI

3. Which networking type did you use?
   (For example: nova-network, Neutron with OpenVSwitch, ...)

   N/A

Logs & Configs
==============

Example Newton trace :

2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher [req-f31f6110-e880-41e8-be1a-2c41ea7fd9ce 75fbc7a6db34480091d2a53e2e20b695 62e53e5e804e49a9890928a5a4846f60 - - -] Exception during message handling: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/disk/by-id/dm-uuid-mpath-360014053aac4f90daef4d76baa773169': Permission denied
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     payload)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5033, in swap_volume
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     resize_to)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4980, in _swap_volume
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.volume_api.unreserve_volume(context, new_volume_id)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4958, in _swap_volume
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     resize_to)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1434, in swap_volume
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self._swap_volume(guest, disk_dev, conf.source_path, resize_to)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1391, in _swap_volume
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     dev.rebase(new_path, copy=True, reuse_ext=True)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 546, in rebase
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self._disk, base, self.REBASE_DEFAULT_BANDWIDTH, flags=flags)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 930, in blockRebase
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self)
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/disk/by-id/dm-uuid-mpath-360014053aac4f90daef4d76baa773169': Permission denied
2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher

** Affects: nova
     Importance: Undecided
     Assignee: Lee Yarwood (lyarwood)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1685185

Title:
  disconnect_volume not called when rebase failures are encountered
  during swap_volume

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========

  At present disconnect_volume is not called when rebase failures are
  encountered during swap_volume. This results in the new volume
  remaining connected to the compute host prior to terminate_connection
  then being called. This can in turn lead to left over devices
  remaining on the compute host for some volume backends such as
  LVM/iSCSI.

  Steps to reproduce
  ==================

  Downstream, the easiest way to reproduce this is via
  https://bugzilla.redhat.com/show_bug.cgi?id=1401755 :

  # sudo setenforce 1
  # nova update-volume ${instance_uuid} \
                       ${attached_NFS_volume_id} \
                       ${unattached_iSCSI_volume_id}

  Upstream, I've been unable to get devstack to even work correctly with
  SELinux in enforcing mode so I've been unable to reproduce the
  rollback this way.

  Expected result
  ===============

  New volume disconnected from compute host.

  Actual result
  =============

  New volume remains connected to compute host.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     N/A

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt + KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     NFS + LVM/iSCSI

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============

  Example Newton trace :

  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher [req-f31f6110-e880-41e8-be1a-2c41ea7fd9ce 75fbc7a6db34480091d2a53e2e20b695 62e53e5e804e49a9890928a5a4846f60 - - -] Exception during message handling: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/disk/by-id/dm-uuid-mpath-360014053aac4f90daef4d76baa773169': Permission denied
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 183, in _dispatch
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 110, in wrapped
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     payload)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 359, in decorated_function
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 328, in decorated_function
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 387, in decorated_function
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5033, in swap_volume
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     resize_to)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4980, in _swap_volume
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.volume_api.unreserve_volume(context, new_volume_id)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4958, in _swap_volume
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     resize_to)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1434, in swap_volume
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self._swap_volume(guest, disk_dev, conf.source_path, resize_to)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1391, in _swap_volume
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     dev.rebase(new_path, copy=True, reuse_ext=True)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 546, in rebase
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     self._disk, base, self.REBASE_DEFAULT_BANDWIDTH, flags=flags)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     rv = execute(f, *args, **kwargs)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     six.reraise(c, e, tb)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     rv = meth(*args, **kwargs)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 930, in blockRebase
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher     if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self)
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/dev/disk/by-id/dm-uuid-mpath-360014053aac4f90daef4d76baa773169': Permission denied
  2017-04-12 09:37:53.744 3077 ERROR oslo_messaging.rpc.dispatcher

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1685185/+subscriptions
Follow ups

[Bug 1685185] Re: disconnect_volume not called when rebase failures are encountered during swap_volume
From: Matt Riedemann, 2017-06-06
[Bug 1685185] Re: disconnect_volume not called when rebase failures are encountered during swap_volume
From: OpenStack Infra, 2017-04-26