yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1244694] Re: [SRU] Creating snapshot fails due to nonexistent temporary directory

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Brian Murray <brian@xxxxxxxxxx>
Date: Fri, 17 Jan 2014 01:31:46 -0000
Reply-to: Bug 1244694 <1244694@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Hello Daniel, or anyone else affected,

Accepted libvirt into saucy-proposed. The package will build now and be
available at
http://launchpad.net/ubuntu/+source/libvirt/1.1.1-0ubuntu8.3 in a few
hours, and then in the -proposed repository.

Please help us by testing this new package.  See
https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to
enable and use -proposed.  Your feedback will aid us getting this update
out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug,
mentioning the version of the package you tested, and change the tag
from verification-needed to verification-done. If it does not fix the
bug for you, please add a comment stating that, and change the tag to
verification-failed.  In either case, details of your testing will help
us make a better decision.

Further information regarding the verification process can be found at
https://wiki.ubuntu.com/QATeam/PerformingSRUVerification .  Thank you in
advance!

** Also affects: libvirt (Ubuntu Saucy)
   Importance: Undecided
       Status: New

** Changed in: libvirt (Ubuntu Saucy)
   Importance: Undecided => High

** Changed in: libvirt (Ubuntu Saucy)
       Status: New => Triaged

** Changed in: libvirt (Ubuntu Saucy)
       Status: Triaged => Fix Committed

** Tags added: verification-needed

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1244694

Title:
  [SRU] Creating snapshot fails due to nonexistent temporary directory

Status in OpenStack Compute (Nova):
  New
Status in “libvirt” package in Ubuntu:
  Fix Released
Status in “libvirt” source package in Saucy:
  Fix Committed

Bug description:
  ---- SRU Justification ---

  [Impact]

  In a libvirt-based OpenStack deployment, Nova fails to snapshot
  instances, failing with error:

  internal error: unable to execute QEMU command 'drive-mirror': Could
  not open
  '/var/lib/nova/instances/snapshots/tmp5DdrIO/236df3e170e64fabaeb3c7601e2d6c47.delta'

  I had originally discovered this bug using the Tempset test suite
  while verifying an unrelated OpenStack SRU, but other users are
  experiencing this in the wild.

  [Test Case]

  Deploy an OpenStack cloud based on Ubuntu Saucy and OpenStack Havana,
  then attempt to snapshot a running instance.  The Tempest integration
  test suite contains a snapshot test case:
  tempest.api.compute.images.test_images_oneserver.ImagesOneServerTestJSON.test_create_delete_image
  [1]

  [Regression Potential]

  The proposed fix is isolated to the libvirt packaging and simply
  appends an additional directory exception to the packages apparmor
  configuration, so that libvirt has appropriate access to the directory
  used during the process of snapshotting an instance.

  
  [1]https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_images_oneserver.py#L92


  --- Original Bug ---

  
  In some cases (not for all instances, just for some) the following error prevents creating the snapshot:

  2013-10-25 14:49:30.724 22980 AUDIT nova.compute.manager [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] [instance: db9c8a72-6ce2-41b7-8f7a-be0be8468667] instance snapshotting
  2013-10-25 14:49:30.944 22980 INFO nova.virt.libvirt.driver [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] [instance: db9c8a72-6ce2-41b7-8f7a-be0be8468667] Beginning live snapshot process
  2013-10-25 14:49:32.006 22980 INFO nova.virt.libvirt.driver [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] [instance: db9c8a72-6ce2-41b7-8f7a-be0be8468667] Snapshot extracted, beginning image upload
  2013-10-25 14:49:32.329 22980 ERROR nova.openstack.common.rpc.amqp [req-6e9326d7-64df-40f7-bc81-190ec5234de2 657f1aca48d24eaf9655e0b77b2bc6d9 35b2b08cc3f44a538cf3535043793a2a] Exception during message handling
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     **args)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     payload)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 243, in decorated_function
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     pass
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 229, in decorated_function
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 319, in decorated_function
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     % image_id, instance=instance)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 309, in decorated_function
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     *args, **kwargs)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2293, in snapshot_instance
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     task_states.IMAGE_SNAPSHOT)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2324, in _snapshot_instance
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     update_task_state)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1397, in snapshot
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     image_format)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1483, in _live_snapshot
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     libvirt.VIR_DOMAIN_BLOCK_REBASE_SHALLOW)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 179, in doit
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 139, in proxy_call
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     rv = execute(f,*args,**kwargs)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 77, in tworker
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     rv = meth(*args,**kwargs)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 646, in blockRebase
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp     if ret == -1: raise libvirtError ('virDomainBlockRebase() failed', dom=self)
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp libvirtError: internal error: unable to execute QEMU command 'drive-mirror': Could not open '/var/lib/nova/instances/snapshots/tmp5DdrIO/236df3e170e64fabaeb3c7601e2d6c47.delta'
  2013-10-25 14:49:32.329 22980 TRACE nova.openstack.common.rpc.amqp

  The directory /var/lib/nova/instances/snapshots/tmp5DdrIO does not exist immidiately after this error shows up.
  Permissions to the parent directory are nova:nova 755, I believe they are correct.

  Package: nova-compute
  Version: 1:2013.2-0ubuntu1~cloud0

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1244694/+subscriptions