yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1584630] [NEW] rebuild don't respect image's block device mapping propertries

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jiajun Liu <iamljj@xxxxxxxxx>
Date: Mon, 23 May 2016 07:51:22 -0000
Reply-to: Bug 1584630 <1584630@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Steps to reproduce
===================

1, launch a boot from volume instance named bfv1
2, attach a volume (blank-volume1) to instance bfv1
3, take a snapshot on bfv1 and a new image bfv1-snap1 created
4, launch another boot from image instance named bfi1
4, rebuild bfi1 from bfv1-snap1 failed
5, error log is as follow

2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     payload)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2709, in rebuild_instance
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     bdms, recreate, on_shared_storage, preserve_ephemeral)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2748, in _do_rebuild_instance_with_claim
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     self._do_rebuild_instance(*args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2857, in _do_rebuild_instance
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     self._rebuild_default_impl(**kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2629, in _rebuild_default_impl
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     block_device_info=new_block_device_info)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2562, in spawn
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     admin_pass=admin_password)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2988, in _create_image
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     instance, size, fallback_from_host)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6427, in _try_fetch_image_cache
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     size=size)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 240, in cache
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     *args, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 835, in create_image
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     self.driver.import_image(base, self.rbd_name)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 313, in import_image
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     utils.execute('rbd', 'import', *args)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return processutils.execute(*cmd, **kwargs)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     cmd=sanitized_cmd)
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Command: rbd import --pool openstack-00 /var/lib/nova/instances/_base/9c73eb8ef590d6639ef72dfd807b4d8c7c4c45d9 cc83293e-afc7-4251-bd89-c8dc1d607ff3_disk --image-format=2 --id admin --conf /etc/ceph/ceph.conf
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Exit code: 25
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stdout: u''
2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stderr: u"2016-05-23 07:40:43.707197 7f9ba29d17c0 -1 asok(0xba8500) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-131697.asok': (13) Permission denied\nrbd: unable to get size of file/block device\n\rImporting image: 0% complete...failed.\nrbd: import failed: (25) Inappropriate ioctl for device\n"

Environment
============
 * os: centos7
 * openstack: stable/liberty
 * storage: ceph rbd

I looked at the master code and it have the same bug.

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Steps to reproduce
  ===================
  
  1, launch a boot from volume instance named bfv1
  2, attach a volume (blank-volume1) to instance bfv1
  3, take a snapshot on bfv1 and a new image bfv1-snap1 created
  4, launch another boot from image instance named bfi1
  4, rebuild bfi1 from bfv1-snap1 failed
  5, error log is as follow
  
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     payload)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2709, in rebuild_instance
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     bdms, recreate, on_shared_storage, preserve_ephemeral)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2748, in _do_rebuild_instance_with_claim
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     self._do_rebuild_instance(*args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2857, in _do_rebuild_instance
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     self._rebuild_default_impl(**kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2629, in _rebuild_default_impl
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     block_device_info=new_block_device_info)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2562, in spawn
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     admin_pass=admin_password)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2988, in _create_image
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     instance, size, fallback_from_host)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6427, in _try_fetch_image_cache
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     size=size)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 240, in cache
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     *args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 835, in create_image
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     self.driver.import_image(base, self.rbd_name)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 313, in import_image
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     utils.execute('rbd', 'import', *args)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return processutils.execute(*cmd, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     cmd=sanitized_cmd)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Command: rbd import --pool openstack-00 /var/lib/nova/instances/_base/9c73eb8ef590d6639ef72dfd807b4d8c7c4c45d9 cc83293e-afc7-4251-bd89-c8dc1d607ff3_disk --image-format=2 --id admin --conf /etc/ceph/ceph.conf
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Exit code: 25
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stdout: u''
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stderr: u"2016-05-23 07:40:43.707197 7f9ba29d17c0 -1 asok(0xba8500) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-131697.asok': (13) Permission denied\nrbd: unable to get size of file/block device\n\rImporting image: 0% complete...failed.\nrbd: import failed: (25) Inappropriate ioctl for device\n"
  
- 
  Environment
  ============
-  * os: centos7
-  * openstack: stable/liberty
+  * os: centos7
+  * openstack: stable/liberty
+  * storage: ceph rbd
  
  I looked at the master code and it have the same bug.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1584630

Title:
  rebuild don't respect image's block device mapping propertries

Status in OpenStack Compute (nova):
  New

Bug description:
  Steps to reproduce
  ===================

  1, launch a boot from volume instance named bfv1
  2, attach a volume (blank-volume1) to instance bfv1
  3, take a snapshot on bfv1 and a new image bfv1-snap1 created
  4, launch another boot from image instance named bfi1
  4, rebuild bfi1 from bfv1-snap1 failed
  5, error log is as follow

  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 142, in inner
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 89, in wrapped
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     payload)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 72, in wrapped
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 350, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     LOG.warning(msg, e, instance=instance)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 323, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 400, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 378, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 366, in decorated_function
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2709, in rebuild_instance
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     bdms, recreate, on_shared_storage, preserve_ephemeral)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2748, in _do_rebuild_instance_with_claim
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     self._do_rebuild_instance(*args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2857, in _do_rebuild_instance
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     self._rebuild_default_impl(**kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2629, in _rebuild_default_impl
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     block_device_info=new_block_device_info)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2562, in spawn
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     admin_pass=admin_password)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2988, in _create_image
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     instance, size, fallback_from_host)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6427, in _try_fetch_image_cache
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     size=size)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 240, in cache
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     *args, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 835, in create_image
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     self.driver.import_image(base, self.rbd_name)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/storage/rbd_utils.py", line 313, in import_image
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     utils.execute('rbd', 'import', *args)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 390, in execute
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     return processutils.execute(*cmd, **kwargs)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 275, in execute
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher     cmd=sanitized_cmd)
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Command: rbd import --pool openstack-00 /var/lib/nova/instances/_base/9c73eb8ef590d6639ef72dfd807b4d8c7c4c45d9 cc83293e-afc7-4251-bd89-c8dc1d607ff3_disk --image-format=2 --id admin --conf /etc/ceph/ceph.conf
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Exit code: 25
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stdout: u''
  2016-05-23 07:40:44.064 136993 ERROR oslo_messaging.rpc.dispatcher Stderr: u"2016-05-23 07:40:43.707197 7f9ba29d17c0 -1 asok(0xba8500) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/var/run/ceph/rbd-131697.asok': (13) Permission denied\nrbd: unable to get size of file/block device\n\rImporting image: 0% complete...failed.\nrbd: import failed: (25) Inappropriate ioctl for device\n"

  Environment
  ============
   * os: centos7
   * openstack: stable/liberty
   * storage: ceph rbd

  I looked at the master code and it have the same bug.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1584630/+subscriptions