yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1379434] Re: Nova unable to download images from glance with rbd backend (juno/rc1)

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: James Page <james.page@xxxxxxxxxx>
Date: Thu, 09 Oct 2014 17:44:34 -0000
Reply-to: Bug 1379434 <1379434@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Adding juno-rc-potential tag, as although this is not technically in
glance, glance_store is a fairly important dependency; Ubuntu has cherry
picked several fixes for ceph rbd support.

** Tags added: juno-rc-potential

** Changed in: nova (Ubuntu)
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1379434

Title:
  Nova unable to download images from glance with rbd backend (juno/rc1)

Status in OpenStack Image Registry and Delivery Service (Glance):
  New
Status in “nova” package in Ubuntu:
  Invalid
Status in “python-glance-store” package in Ubuntu:
  New

Bug description:
  2014-10-09 17:00:41.356 9242 AUDIT nova.compute.manager [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Starting instance...
  2014-10-09 17:00:41.435 9242 WARNING nova.compute.resource_tracker [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Host field should not be set on the instance until resources have been claimed.
  2014-10-09 17:00:41.435 9242 WARNING nova.compute.resource_tracker [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Node field should not be set on the instance until resources have been claimed.
  2014-10-09 17:00:41.437 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Attempting claim: memory 512 MB, disk 1 GB
  2014-10-09 17:00:41.437 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Total memory: 3952 MB, used: 512.00 MB
  2014-10-09 17:00:41.438 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] memory limit: 5928.00 MB, free: 5416.00 MB
  2014-10-09 17:00:41.438 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Total disk: 9 GB, used: 0.00 GB
  2014-10-09 17:00:41.439 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] disk limit not specified, defaulting to unlimited
  2014-10-09 17:00:41.449 9242 AUDIT nova.compute.claims [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Claim successful
  2014-10-09 17:00:41.550 9242 INFO nova.scheduler.client.report [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] Compute_service record updated for ('juju-t-machine-15', 'juju-t-machine-15.openstacklocal')
  2014-10-09 17:00:41.871 9242 INFO nova.virt.libvirt.driver [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Creating image
  2014-10-09 17:00:41.929 9242 ERROR glanceclient.common.http [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 ] Request returned failure status 500.
  2014-10-09 17:00:41.930 9242 ERROR nova.compute.manager [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Instance failed to spawn
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Traceback (most recent call last):
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2231, in _build_resources
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     yield resources
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2101, in _build_and_run_instance
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     block_device_info=block_device_info)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2613, in spawn
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     admin_pass=admin_password)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3003, in _create_image
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     project_id=instance['project_id'])
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 208, in cache
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     *args, **kwargs)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 451, in create_image
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     prepare_template(target=base, max_size=size, *args, **kwargs)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 272, in inner
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     return f(*args, **kwargs)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 198, in fetch_func_sync
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     fetch_func(target=target, *args, **kwargs)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 452, in fetch_image
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     max_size=max_size)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 79, in fetch_to_raw
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     max_size=max_size)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 73, in fetch
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     IMAGE_API.download(context, image_href, dest_path=path)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/image/api.py", line 178, in download
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     dst_path=dest_path)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 359, in download
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     _reraise_translated_image_exception(image_id)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 357, in download
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     image_chunks = self._client.call(context, 1, 'data', image_id)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 232, in call
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     return getattr(client.images, method)(*args, **kwargs)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 142, in data
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     % urlparse.quote(str(image_id)))
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 253, in get
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     return self._request('GET', url, **kwargs)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 221, in _request
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]     raise exc.from_response(resp, resp.content)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
  2014-10-09 17:00:41.930 9242 TRACE nova.compute.manager [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa]
  2014-10-09 17:00:41.932 9242 AUDIT nova.compute.manager [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Terminating instance
  2014-10-09 17:00:41.941 9242 WARNING nova.virt.libvirt.driver [-] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] During wait destroy, instance disappeared.
  2014-10-09 17:00:42.137 9242 INFO nova.virt.libvirt.driver [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Deleting instance files /var/lib/nova/instances/826a1d2d-a341-45c2-b070-c20f30b4d5aa_del
  2014-10-09 17:00:42.138 9242 INFO nova.virt.libvirt.driver [req-7b7a713b-087a-46ce-8c6f-1e287abf3c99 None] [instance: 826a1d2d-a341-45c2-b070-c20f30b4d5aa] Deletion of /var/lib/nova/instances/826a1d2d-a341-45c2-b070-c20f30b4d5aa_del complete

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1379434/+subscriptions