yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1300086] [NEW] Snapshot creation fails with ceph and cephx enabled

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alexander Stellwag <1300086@xxxxxxxxxxxxxxxxxx>
Date: Mon, 31 Mar 2014 08:30:33 -0000
Reply-to: Bug 1300086 <1300086@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Using ceph and authentication (cephx), snapshot creation fails here:

2014-03-27 15:56:18.262 38630 INFO nova.virt.libvirt.driver [req-3a609d8a-13ef-4d26-b01c-ecb01b65f686 58cc5
e4467054008b0c3d568cbd003a8 1a48884f26d748a9991eea625859ff5d] [instance: 9dffc6f7-95c6-4fe7-98d7-4e3f10bdf2
b6] Snapshot extracted, beginning image upload
2014-03-27 15:56:18.525 38630 ERROR nova.openstack.common.rpc.amqp [req-3a609d8a-13ef-4d26-b01c-ecb01b65f68
6 58cc5e4467054008b0c3d568cbd003a8 1a48884f26d748a9991eea625859ff5d] Exception during message handling
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     **args)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(c
txt, **kwargs)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/compute/manager.py", line 353, in decorated_function
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args
, **kwargs)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/exception.py", line 90, in wrapped
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     payload)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/exception.py", line 73, in wrapped
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/compute/manager.py", line 243, in decorated_function
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     pass
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/compute/manager.py", line 229, in decorated_function
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args
, **kwargs)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/compute/manager.py", line 271, in decorated_function
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/compute/manager.py", line 258, in decorated_function
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args
, **kwargs)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/compute/manager.py", line 319, in decorated_function
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     % image_id, instance=instance)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/compute/manager.py", line 309, in decorated_function
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     *args, **kwargs)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/compute/manager.py", line 2310, in snapshot_instance
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     task_states.IMAGE_SNAPSHOT)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
/nova/2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     update_task_state)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1410, in snapshot
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     snapshot_backend.snapshot_extract(out_path, image_format)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 560, in snapshot_extract
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     images.convert_image(snap, target, out_format)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 179, in convert_image
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     utils.execute(*cmd, run_as_root=run_as_root)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 177, in execute
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 178, in execute
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Command: qemu-img convert -O raw rbd:volumes/instance-000003b0_disk /var/lib/nova/instances/snapshots/tmpyh4qLE/8bd0281307c94de8a8621eb1e31ef020
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Exit code: 1
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Stdout: ''
2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Stderr: "qemu-img: error connecting\nqemu-img: Could not open 'rbd:volumes/instance-000003b0_disk': No such file or directory\nqemu-img: Could not open 'rbd:volumes/instance-000003b0_disk'\n"compute/manager.py", line 2341, in _snapshot_instance

The problem ist that 'qemu-img' is called without the proper rbd_user.
The attached patch seems to fix this.

We're on ubuntu 12.04 here and using the havana-packages from UCA.

** Affects: nova
     Importance: Undecided
         Status: New

** Patch added: "nova_cephx.patch"
   https://bugs.launchpad.net/bugs/1300086/+attachment/4055136/+files/nova_cephx.patch

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1300086

Title:
  Snapshot creation fails with ceph and cephx enabled

Status in OpenStack Compute (Nova):
  New

Bug description:
  Using ceph and authentication (cephx), snapshot creation fails here:

  2014-03-27 15:56:18.262 38630 INFO nova.virt.libvirt.driver [req-3a609d8a-13ef-4d26-b01c-ecb01b65f686 58cc5
  e4467054008b0c3d568cbd003a8 1a48884f26d748a9991eea625859ff5d] [instance: 9dffc6f7-95c6-4fe7-98d7-4e3f10bdf2
  b6] Snapshot extracted, beginning image upload
  2014-03-27 15:56:18.525 38630 ERROR nova.openstack.common.rpc.amqp [req-3a609d8a-13ef-4d26-b01c-ecb01b65f68
  6 58cc5e4467054008b0c3d568cbd003a8 1a48884f26d748a9991eea625859ff5d] Exception during message handling
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     **args)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(c
  txt, **kwargs)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/compute/manager.py", line 353, in decorated_function
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args
  , **kwargs)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/exception.py", line 90, in wrapped
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     payload)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/exception.py", line 73, in wrapped
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/compute/manager.py", line 243, in decorated_function
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     pass
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/compute/manager.py", line 229, in decorated_function
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args
  , **kwargs)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/compute/manager.py", line 271, in decorated_function
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/compute/manager.py", line 258, in decorated_function
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args
  , **kwargs)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/compute/manager.py", line 319, in decorated_function
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     % image_id, instance=instance)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/compute/manager.py", line 309, in decorated_function
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     *args, **kwargs)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/compute/manager.py", line 2310, in snapshot_instance
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     task_states.IMAGE_SNAPSHOT)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages
  /nova/2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     update_task_state)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1410, in snapshot
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     snapshot_backend.snapshot_extract(out_path, image_format)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 560, in snapshot_extract
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     images.convert_image(snap, target, out_format)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 179, in convert_image
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     utils.execute(*cmd, run_as_root=run_as_root)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 177, in execute
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     return processutils.execute(*cmd, **kwargs)
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/processutils.py", line 178, in execute
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp     cmd=' '.join(cmd))
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp ProcessExecutionError: Unexpected error while running command.
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Command: qemu-img convert -O raw rbd:volumes/instance-000003b0_disk /var/lib/nova/instances/snapshots/tmpyh4qLE/8bd0281307c94de8a8621eb1e31ef020
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Exit code: 1
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Stdout: ''
  2014-03-27 15:56:18.525 38630 TRACE nova.openstack.common.rpc.amqp Stderr: "qemu-img: error connecting\nqemu-img: Could not open 'rbd:volumes/instance-000003b0_disk': No such file or directory\nqemu-img: Could not open 'rbd:volumes/instance-000003b0_disk'\n"compute/manager.py", line 2341, in _snapshot_instance

  The problem ist that 'qemu-img' is called without the proper rbd_user.
  The attached patch seems to fix this.

  We're on ubuntu 12.04 here and using the havana-packages from UCA.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1300086/+subscriptions
Follow ups

[Bug 1300086] Re: Snapshot creation fails with ceph and cephx enabled
From: Josh Durgin, 2014-09-30
[Bug 1300086] [NEW] Snapshot creation fails with ceph and cephx enabled
From: Alexander Stellwag, 2014-03-31
References

[Bug 1300086] [NEW] Snapshot creation fails with ceph and cephx enabled
From: Alexander Stellwag, 2014-03-31