yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1198566] Re: expired image location url cause glance client errors

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Alan Pevec <1198566@xxxxxxxxxxxxxxxxxx>
Date: Thu, 04 Dec 2014 21:43:16 -0000
Reply-to: Bug 1198566 <1198566@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Also affects: glance/juno
   Importance: Undecided
       Status: New

** Changed in: glance/juno
       Status: New => Fix Committed

** Changed in: glance/juno
    Milestone: None => 2014.2.1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1198566

Title:
  expired image location url cause glance client errors

Status in OpenStack Image Registry and Delivery Service (Glance):
  Fix Committed
Status in Glance juno series:
  Fix Committed

Bug description:
  We have a multi-node Openstack cluster running Folsom 2012.2.3 on
  Ubuntu 12.04. A few days ago we added a new compute node, and found
  that we were unable to launch new instances from a pre-existing Ubuntu
  Server 12.04 LTS image stored in glance. Each spawning attempt would
  deposit a glance client exception (shown below) in the compute node's
  nova-compute.log.

  After quite a lot of investigation, I found that the original
  --location URL (used during "glance image-create") of the Ubuntu
  Server image had gone out of date. This was evidently causing a
  BadStoreUri exception on the glance server during instance spawning,
  resulting in a 500 error being returned to our new compute node's
  glance client. I was able to resolve the problem by re-importing an
  Ubuntu Server 12.04 LTS image from a working mirror.

  Improved error logging would have saved us hours of troubleshooting.

  2013-06-27 21:19:24 ERROR nova.compute.manager [req-f8d7c23a-e8ad-4059-bea4-4fc588a6afe0 9d8968d3f17f4697aaf923c14651ce7b e5fb3c6db0db4e9c86d0301005e2e5bb] [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] Instance failed to spawn
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] Traceback (most recent call last):
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 756, in _spawn
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     block_device_info)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 117, in wrapped
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     temp_level, payload)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     self.gen.next()
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 92, in wrapped
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     return f(*args, **kw)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1099, in spawn
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     admin_pass=admin_password)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1365, in _create_image
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     project_id=instance['project_id'])
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 131, in cache
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     *args, **kwargs)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 178, in create_image
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     prepare_template(target=base, *args, **kwargs)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 795, in inner
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     retval = f(*args, **kwargs)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 122, in call_if_not_exists
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     fetch_func(target=target, *args, **kwargs)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/utils.py", line 440, in fetch_image
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     images.fetch_to_raw(context, image_id, target, user_id, project_id)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 87, in fetch_to_raw
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     fetch(context, image_href, path_tmp, user_id, project_id)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 82, in fetch
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     image_service.download(context, image_id, image_file)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 222, in download
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     _reraise_translated_image_exception(image_id)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 220, in download
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     image_chunks = self._client.call(context, 1, 'data', image_id)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/nova/image/glance.py", line 138, in call
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     return getattr(client.images, method)(*args, **kwargs)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 108, in data
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     resp, body = self.api.raw_request('GET', '/v1/images/%s' % image_id)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 191, in raw_request
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     return self._http_request(url, method, **kwargs)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]   File "/usr/lib/python2.7/dist-packages/glanceclient/common/http.py", line 158, in _http_request
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]     raise exc.from_response(resp)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d] HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
  2013-06-27 21:19:24 5290 TRACE nova.compute.manager [instance: 1cdd84ad-ba1b-4e5c-8711-8e2c91b48c3d]

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1198566/+subscriptions