yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1659220] Re: Attaching volume after taking snapshot fails silently

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Fri, 28 Jul 2017 12:51:12 -0000
Reply-to: Bug 1659220 <1659220@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Managing the AppArmor config is currently beyond scope for Nova, thus
marking this as Opinion. It would be good if those rules from the
distros were able to do this better.

If you wanted to put that in scope for Nova, that would require a Nova
spec - https://specs.openstack.org/openstack/nova-specs/readme.html

** Changed in: nova
       Status: New => Opinion

** Changed in: nova
   Importance: Undecided => Wishlist

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1659220

Title:
  Attaching volume after taking snapshot fails silently

Status in OpenStack Compute (nova):
  Opinion

Bug description:
  Description
  ===========
  You have an instance and a volume and attach the volume against this instance everything works fine. But if you take a snapshot of this volume (detached) and try to attach again Horizon tells you everything was ok but the volume stays in the available state and isn't available in the instance.

  The actual problem seems to come from libvirt as it doesn't create the
  required AppArmor rules in the
  /etc/apparmor.d/libvirt/libvirt-<UUID>.files

  Steps to reproduce
  ==================
  * create an instance
  * create an volume
  * snapshot the volume
  * attach the volume

  Expected result
  ===============
  Volume should be attached to the instance.

  Actual result
  =============
  Volume stays available not attached.

  Environment
  ===========
  Ubuntu 16.04 with Newton via Cloud Archive.

  1. Exact version of OpenStack you are running. See the following
  ii  nova-common                        2:14.0.1-0ubuntu1~cloud0            all
  ii  nova-compute                       2:14.0.1-0ubuntu1~cloud0            all
  ii  nova-compute-kvm                   2:14.0.1-0ubuntu1~cloud0            all
  ii  nova-compute-libvirt               2:14.0.1-0ubuntu1~cloud0            all
  ii  python-nova                        2:14.0.1-0ubuntu1~cloud0            all
  ii  python-novaclient                  2:6.0.0-0ubuntu1~cloud0             all

  2. Which hypervisor did you use?
  ii  libvirt-bin                        1.3.1-1ubuntu10.6                   amd64
  ii  libvirt0:amd64                     1.3.1-1ubuntu10.6                   amd64
  ii  python-libvirt                     1.3.1-1ubuntu1                      amd64

  ii  ipxe-qemu                          1.0.0+git-20150424.a25a16d-1ubuntu1 all
  ii  qemu-block-extra:amd64             1:2.5+dfsg-5ubuntu10.6              amd64
  ii  qemu-kvm                           1:2.5+dfsg-5ubuntu10.6              amd64
  ii  qemu-system-common                 1:2.5+dfsg-5ubuntu10.6              amd64
  ii  qemu-system-x86                    1:2.5+dfsg-5ubuntu10.6              amd64
  ii  qemu-utils                         1:2.5+dfsg-5ubuntu10.6              amd64

  2. Which storage type did you use?
  ii  glusterfs-client                   3.7.6-1ubuntu1                      amd64
  ii  glusterfs-common                   3.7.6-1ubuntu1                      amd64

  3. Which networking type did you use?
  ii  neutron-common                     2:9.0.0-0ubuntu1.16.10.2~cloud0     all
  ii  neutron-linuxbridge-agent          2:9.0.0-0ubuntu1.16.10.2~cloud0     all
  ii  neutron-plugin-linuxbridge-agent   2:9.0.0-0ubuntu1.16.10.2~cloud0     all
  ii  python-neutron                     2:9.0.0-0ubuntu1.16.10.2~cloud0     all
  ii  python-neutron-fwaas               1:9.0.0-0ubuntu1~cloud0             all
  ii  python-neutron-lib                 0.4.0-0ubuntu1~cloud0               all
  ii  python-neutronclient               1:6.0.0-0ubuntu1~cloud0             all

  Logs & Configs
  ==============

  kern.log
  Jan 25 09:29:03 ext kernel: [6527389.763944] audit: type=1400 audit(1485332943.463:125): apparmor="DENIED" operation="open" profile="libvirt-c9c9ec47-f4e7-4065-9b45-947d1c9efcd3" name="/var/lib/nova/mnt/974fd68b172ee6f649b9ada09de2edf4/volume-fb44d9f4-4247-44e1-98a9-14bde7582ab0" pid=30968 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=113 ouid=113
  Jan 25 09:29:03 ext kernel: [6527389.763999] audit: type=1400 audit(1485332943.463:126): apparmor="DENIED" operation="open" profile="libvirt-c9c9ec47-f4e7-4065-9b45-947d1c9efcd3" name="/var/lib/nova/mnt/974fd68b172ee6f649b9ada09de2edf4/volume-fb44d9f4-4247-44e1-98a9-14bde7582ab0" pid=30968 comm="qemu-system-x86" requested_mask="r" denied_mask="r" fsuid=113 ouid=113

  libvirt-bin:
  Jan 25 09:39:57 m2r1 libvirtd[31600]: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'

  nova-compute:
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [req-e164a413-2688-4906-93e1-814da66e4b87 78009cee2c13413db3f15ef242e9073c 6c8b3ce916864989ba15a76dfe88e0a9 - - -] [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] Failed to attach volume at mountpoint: /dev/vdb
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] Traceback (most recent call last):
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1146, in attach_volume
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     guest.attach_device(conf, persistent=True, live=live)
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 295, in attach_device
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     self._domain.attachDeviceFlags(device_xml, flags=flags)
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     rv = execute(f, *args, **kwargs)
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     six.reraise(c, e, tb)
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     rv = meth(*args, **kwargs)
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 560, in attachDeviceFlags
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
  2017-01-25 09:39:57.287 4091 ERROR nova.virt.libvirt.driver [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] 
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [req-e164a413-2688-4906-93e1-814da66e4b87 78009cee2c13413db3f15ef242e9073c 6c8b3ce916864989ba15a76dfe88e0a9 - - -] [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] Driver failed to attach volume fb44d9f4-4247-44e1-98a9-14bde7582ab0 at /dev/vdb
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] Traceback (most recent call last):
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     device_type=self['device_type'], encryption=encryption)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1157, in attach_volume
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     self._disconnect_volume(connection_info, disk_dev)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     self.force_reraise()
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     six.reraise(self.type_, self.value, self.tb)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1146, in attach_volume
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     guest.attach_device(conf, persistent=True, live=live)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 295, in attach_device
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     self._domain.attachDeviceFlags(device_xml, flags=flags)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     rv = execute(f, *args, **kwargs)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     six.reraise(c, e, tb)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     rv = meth(*args, **kwargs)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 560, in attachDeviceFlags
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
  2017-01-25 09:39:57.804 4091 ERROR nova.virt.block_device [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] 
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [req-e164a413-2688-4906-93e1-814da66e4b87 78009cee2c13413db3f15ef242e9073c 6c8b3ce916864989ba15a76dfe88e0a9 - - -] [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] Failed to attach fb44d9f4-4247-44e1-98a9-14bde7582ab0 at /dev/vdb
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] Traceback (most recent call last):
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4680, in _attach_volume
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     do_check_attach=False, do_driver_attach=True)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 49, in wrapped
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     ret_val = method(obj, context, *args, **kwargs)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 283, in attach
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     connector)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     self.force_reraise()
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     six.reraise(self.type_, self.value, self.tb)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     device_type=self['device_type'], encryption=encryption)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1157, in attach_volume
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     self._disconnect_volume(connection_info, disk_dev)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     self.force_reraise()
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     six.reraise(self.type_, self.value, self.tb)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1146, in attach_volume
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     guest.attach_device(conf, persistent=True, live=live)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 295, in attach_device
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     self._domain.attachDeviceFlags(device_xml, flags=flags)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     rv = execute(f, *args, **kwargs)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     six.reraise(c, e, tb)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     rv = meth(*args, **kwargs)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 560, in attachDeviceFlags
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3]     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
  2017-01-25 09:39:58.248 4091 ERROR nova.compute.manager [instance: c9c9ec47-f4e7-4065-9b45-947d1c9efcd3] 
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server [req-e164a413-2688-4906-93e1-814da66e4b87 78009cee2c13413db3f15ef242e9073c 6c8b3ce916864989ba15a76dfe88e0a9 - - -] Exception during message handling
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 75, in wrapped
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 66, in wrapped
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 216, in decorated_function
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 204, in decorated_function
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4670, in attach_volume
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     do_attach_volume(context, instance, driver_bdm)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4668, in do_attach_volume
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     bdm.destroy()
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4665, in do_attach_volume
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     return self._attach_volume(context, instance, driver_bdm)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4688, in _attach_volume
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     self.volume_api.unreserve_volume(context, bdm.volume_id)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4680, in _attach_volume
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     do_check_attach=False, do_driver_attach=True)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 49, in wrapped
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     ret_val = method(obj, context, *args, **kwargs)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 283, in attach
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     connector)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 274, in attach
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     device_type=self['device_type'], encryption=encryption)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1157, in attach_volume
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     self._disconnect_volume(connection_info, disk_dev)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1146, in attach_volume
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     guest.attach_device(conf, persistent=True, live=live)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 295, in attach_device
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     self._domain.attachDeviceFlags(device_xml, flags=flags)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     rv = execute(f, *args, **kwargs)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     six.reraise(c, e, tb)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     rv = meth(*args, **kwargs)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 560, in attachDeviceFlags
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server     if ret == -1: raise libvirtError ('virDomainAttachDeviceFlags() failed', dom=self)
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server libvirtError: internal error: unable to execute QEMU command 'device_add': Property 'virtio-blk-device.drive' can't find value 'drive-virtio-disk1'
  2017-01-25 09:39:58.492 4091 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1659220/+subscriptions
References

[Bug 1659220] [NEW] Attaching volume after taking snapshot fails silently
From: Paul Seidler, 2017-01-25