yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1370387] Re: Fail to fetch non-raw image on image_backend=RBD

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Yaguang Tang <yaguang.tang@xxxxxxxxxxxxx>
Date: Fri, 21 Nov 2014 05:50:23 -0000
Reply-to: Bug 1370387 <1370387@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
*** This bug is a duplicate of bug 1371040 ***
    https://bugs.launchpad.net/bugs/1371040

The glance traceback shows that it's a bug in glance_store. This is actually a duplicate of https://bugs.launchpad.net/glance/+bug/1371040. 
Please update your glance store to  >=0.1.9 which fixes this bug.

** This bug has been marked a duplicate of bug 1371040
   glance store rbd driver get method returns tupe more than basic driver defines

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1370387

Title:
  Fail to fetch non-raw image on image_backend=RBD

Status in OpenStack Compute (Nova):
  Confirmed

Bug description:
  Using the image_backend RBD and booting a non-raw image results in a fallback to fetch_to_raw function where the goal is to download the qcow2 image on the compute, convert it into raw and import it into ceph.
  Fetching the image fails with the following errors:

  On the nova-compute logs:

  2014-09-16 10:16:23.061 ERROR nova.compute.manager [req-4c42af30-bc04-4648-bd12-14419031be80 admin admin] [instance: 18d8659a-0247-4472-97be-9c6c9007689b] Instance failed to spaw
  n
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b] Traceback (most recent call last):
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/compute/manager.py", line 2203, in _build_resourc
  es
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     yield resources
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/compute/manager.py", line 2082, in _build_and_run
  _instance
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     block_device_info=block_device_info)
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2598, in spawn
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     admin_pass=admin_password)
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2974, in _create_im
  age
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     project_id=instance['project_id'])2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 206, in cache2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     *args, **kwargs)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 396, in create_image2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     prepare_template(target=base, max_size=size, *args, **kwargs)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/openstack/common/lockutils.py", line 270, in inner2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     return f(*args, **kwargs)
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 196, in fetch_func_sync
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     fetch_func(target=target, *args, **kwargs)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/virt/libvirt/utils.py", line 452, in fetch_image
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     max_size=max_size)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/virt/images.py", line 73, in fetch_to_raw
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     max_size=max_size)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/virt/images.py", line 67, in fetch
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     IMAGE_API.download(context, image_href, dest_path=path)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/image/api.py", line 178, in download
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     dst_path=dest_path)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/image/glance.py", line 352, in download
  2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     _reraise_translated_image_exception(image_id)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/image/glance.py", line 350, in download2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     image_chunks = self._client.call(context, 1, 'data', image_id)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/nova/nova/image/glance.py", line 225, in call2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     return getattr(client.images, method)(*args, **kwargs)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/python-glanceclient/glanceclient/v1/images.py", line 142, in data2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     % urlparse.quote(str(image_id)))2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/python-glanceclient/glanceclient/common/http.py", line 253, in get2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     return self._request('GET', url, **kwargs)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]   File "/opt/stack/python-glanceclient/glanceclient/common/http.py", line 221, in _request2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]     raise exc.from_response(resp, resp.content)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b] HTTPInternalServerError: HTTPInternalServerError (HTTP 500)2014-09-16 10:16:23.061 TRACE nova.compute.manager [instance: 18d8659a-0247-4472-97be-9c6c9007689b]

  
  On the glance-api logs:

  2014-09-16 10:41:43.728 19722 INFO glance.wsgi.server [-] Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 433, in handle_one_response
      result = self.application(self.environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/glance/glance/common/wsgi.py", line 394, in __call__
      response = req.get_response(self.application)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
      application, catch_exc_info=False)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
      app_iter = application(self.environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/osprofiler/web.py", line 106, in __call__
      return request.get_response(self.application)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
      application, catch_exc_info=False)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
      app_iter = application(self.environ, start_response)
    File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 646, in __call__
      return self._call_app(env, start_response)
    File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py", line 624, in _call_app
      return self._app(env, _fake_start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/glance/glance/common/wsgi.py", line 394, in __call__
      response = req.get_response(self.application)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
      application, catch_exc_info=False)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
      app_iter = application(self.environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/glance/glance/common/wsgi.py", line 394, in __call__
      response = req.get_response(self.application)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
      application, catch_exc_info=False)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
      app_iter = application(self.environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/glance/glance/common/wsgi.py", line 394, in __call__
      response = req.get_response(self.application)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
      application, catch_exc_info=False)
    File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
      app_iter = application(self.environ, start_response)
    File "/usr/lib/python2.7/dist-packages/paste/urlmap.py", line 206, in __call__
      return app(environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
      response = self.app(environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
      return resp(environ, start_response)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
      resp = self.call_func(req, *args, **self.kwargs)
    File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
      return self.func(req, *args, **kwargs)
    File "/opt/stack/glance/glance/common/wsgi.py", line 683, in __call__
      request, **action_args)
    File "/opt/stack/glance/glance/common/wsgi.py", line 707, in dispatch
      return method(*args, **kwargs)
    File "/opt/stack/glance/glance/api/v1/images.py", line 494, in show
      image_meta['location'])
    File "/opt/stack/glance/glance/api/v1/images.py", line 456, in _get_from_store
      image_data, image_size = src_store.get(loc, context=context)
  ValueError: too many values to unpack

  
  Using glance v2 should work, however it used to work on v1.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1370387/+subscriptions
References

[Bug 1370387] [NEW] Fail to fetch non-raw image on image_backend=RBD
From: Sébastien Han, 2014-09-17