yahoo-eng-team team mailing list archive

[Sean Dague <sean@xxxxxxxxx>]

** Changed in: nova/liberty
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1470437

Title:
  ImageCacheManager raises Permission denied error on nova compute in
  race condition

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) liberty series:
  Fix Released

Bug description:
  ImageCacheManager raises Permission denied error on nova compute in
  race condition

  While creating an instance snapshot nova calls guest.launch method
  from libvirt driver which changes the base file permissions and
  updates base file user from openstack to libvirt-qemu (in case of
  qcow2 image backend). In race condition when ImageCacheManager is
  trying to update last access time of this base file and guest.launch
  is called by instance snapshot just before updating the access time,
  ImageCacheManager raise Permission denied error in nova compute for
  os.utime().

  Steps to reproduce:
  1. Configure image_cache_manager_interval=120 in nova.conf and use qcow2 image backend.
  2. Add a sleep for 60 sec in _handle_base_image method of libvirt.imagecache just before calling os.utime().
  3. Restart nova services.
  4. Create an instance using image.
  $ nova boot --image 5e1659aa-6d38-44e8-aaa3-4217337436c0 --flavor 1 instance-1
  5. Check that instance is in active state.
  6. Go to the n-cpu screen and check imagecache manager logs at the point it waits to execute sleep statement added in step #2.
  7. Send instance snapshot request when imagecache manger is waiting to execute sleep.
  $ nova image-create 19c7900b-73d5-4c2e-b129-5e2a6b13f396 instance-1-snap
  8. instance snapshot request updates the base file owner to libvirt-qemu by calling guest.launch method from libvirt driver.
  9. Now when imagecache manger comes out from sleep and executes os.utime it raise following Permission denied error in nova compute.

  2015-07-01 01:51:46.794 ERROR nova.openstack.common.periodic_task [req-a03fa45f-ffb9-48dd-8937-5b0414c6864b None None] Error during ComputeManager._run_image_cache_manager_pass
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task Traceback(most recent call last):
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task   File "/opt/stack/nova/nova/openstack/common/periodic_task.py", line 224, in run_periodic_tasks
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task     task(self, context)
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task   File "/opt/stack/nova/nova/compute/manager.py", line 6177, in _run_image_cache_manager_pass
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task     self.driver.manage_image_cache(context, filtered_instances)
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6252, in manage_image_cache
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task     self.image_cache_manager.update(context, all_instances)
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task   File "/opt/stack/nova/nova/virt/libvirt/imagecache.py", line 668, in update
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task
  self._age_and_verify_cached_images(context, all_instances, base_dir)
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task   File "/opt/stack/nova/nova/virt/libvirt/imagecache.py", line 598, in _age_and_verify_cached_images
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task     self._handle_base_image(img, base_file)
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task   File "/opt/stack/nova/nova/virt/libvirt/imagecache.py", line 570, in _handle_base_image
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task     os.utime(base_file, None)
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task OSError:[Errno 13] Permission denied: '/opt/stack/data/nova/instances/_base/8d2c340dcce68e48a75457b1e91457feed27aef5'
  2015-07-01 01:51:46.794 TRACE nova.openstack.common.periodic_task

  Expected result: guest.launch should not update the base file
  permissions and owner to libvirt-qemu.  Base file owner should remain
  unchanged.

  Actual result: Libvirt is updating the base file owner which causes
  permission issues in nova.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1470437/+subscriptions