yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1765309] Re: 500 error on image-download if image is not present at any location

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Erno Kuvaja <jokke@xxxxxx>
Date: Fri, 20 Apr 2018 09:52:58 -0000
Reply-to: Bug 1765309 <1765309@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
So if you look at the code there that you modified to introduce this
bug, you are bypassing the try..catch block that will set the exception
that would be raised at the failure causing it trying to raise None
which obviously will fail.

** Changed in: glance
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1765309

Title:
  500 error on image-download if image is not present at any location

Status in Glance:
  Invalid

Bug description:
  As of now for downloading the image it iterates through all the
  locations and if data (image) is not present at any of the location in
  the end it Logs the error message as "Glance tried all active
  locations to get data for image <image-id> but all have failed."
  Further it fails with 500 error.

  This issue can occur either if image is not available in any of the
  location or any error occured during the download process.

  NOTE:
  To reproduce this issue I have manually added 'continue' statement at [1] to assume that image is not available at any of the location.
  [1] https://github.com/openstack/glance/blob/master/glance/location.py#L470

  Steps to reproduce:
  1. Modify glance/location.py as mentioned in NOTE
  2. Restart glance-api service
  3. Download image using:
     $ glance image-download <image-id> --file downloaded_image 

  Expected Result:
  Image data should be downloaded and saved in 'downloaded_image' file

  Actual Result:
  glance image-download 780ffe26-b95a-4b6f-b2c0-c38803bed73a --file abcdUnable to download image '780ffe26-b95a-4b6f-b2c0-c38803bed73a'. (502 Proxy Error: Proxy Error: The proxy server received an invalid: response from an upstream server.: The proxy server could not handle the request GET&nbsp;/image/v2/images/780ffe26-b95a-4b6f-b2c0-c38803bed73a/file.: Reason: Error reading from remote server: Apache/2.4.18 (Ubuntu) Server at 192.168.0.6 Port 80 (HTTP 502))

  
  Glance API logs:
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance.location [None req-cfca70d4-d9a7-495f-9762-0fb2a8eee769 admin admin] Glance tried all active locations to get data for image 780ffe26-b95a-4b6f-b2c0-c38803bed73a but all have failed.
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: CRITICAL glance [None req-cfca70d4-d9a7-495f-9762-0fb2a8eee769 admin admin] Unhandled error: TypeError: 'ImageProxy' object is not callable
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance Traceback (most recent call last):
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     resp = self.call_func(req, *args, **self.kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return self.func(req, *args, **kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     response = req.get_response(self.application)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     application, catch_exc_info=False)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     app_iter = application(self.environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     resp = self.call_func(req, *args, **self.kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return self.func(req, *args, **kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     response = req.get_response(self.application)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     application, catch_exc_info=False)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     app_iter = application(self.environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     resp = self.call_func(req, *args, **self.kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return self.func(req, *args, **kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     response = req.get_response(self.application)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     application, catch_exc_info=False)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     app_iter = application(self.environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     resp = self.call_func(req, *args, **self.kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return self.func(req, *args, **kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/opt/stack/glance/glance/common/wsgi.py", line 809, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     response = req.get_response(self.application)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     application, catch_exc_info=False)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     app_iter = application(self.environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     resp = self.call_func(req, *args, **self.kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return self.func(req, *args, **kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/osprofiler/web.py", line 119, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return request.get_response(self.application)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     application, catch_exc_info=False)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     app_iter = application(self.environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     resp = self.call_func(req, *args, **self.kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return self.func(req, *args, **kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 337, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     response = req.get_response(self._app)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     application, catch_exc_info=False)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     app_iter = application(self.environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     resp = self.call_func(req, *args, **self.kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return self.func(req, *args, **kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/opt/stack/glance/glance/common/wsgi.py", line 809, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     response = req.get_response(self.application)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     application, catch_exc_info=False)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     app_iter = application(self.environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     resp = self.call_func(req, *args, **self.kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return self.func(req, *args, **kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/opt/stack/glance/glance/common/wsgi.py", line 809, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     response = req.get_response(self.application)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     application, catch_exc_info=False)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     app_iter = application(self.environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 131, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     resp = self.call_func(req, *args, **self.kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 196, in call_func
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return self.func(req, *args, **kwargs)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/opt/stack/glance/glance/common/wsgi.py", line 809, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     response = req.get_response(self.application)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1327, in send
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     application, catch_exc_info=False)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1291, in call_application
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     app_iter = application(self.environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return app(environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return resp(environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     response = self.app(environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return resp(environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 145, in __call__
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance     return resp(environ, start_response)
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance TypeError: 'ImageProxy' object is not callable
  Apr 19 06:37:58 signature devstack@g-api.service[5419]: ERROR glance

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1765309/+subscriptions
References

[Bug 1765309] [NEW] 500 error on image-download if image is not present at any location
From: Abhishek Kekane, 2018-04-19