yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1700881] Re: Not found image after uploading image via volumes.upload_to_image

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Fri, 30 Jun 2017 03:10:55 -0000
Reply-to: Bug 1700881 <1700881@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This is probably what caused the regression:

https://review.openstack.org/#/c/477038/

** No longer affects: cinder

** Also affects: cinder
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1700881

Title:
  Not found image after uploading image via volumes.upload_to_image

Status in Cinder:
  New
Status in Glance:
  New

Bug description:
  In recently rally test, sometimes we have uploaded the image, and the
  volume status was AVAILABLE, BUT image.get can not find this image.

  report error:
  http://logs.openstack.org/40/470240/4/check/gate-rally-dsvm-rally-cinder/c49be3d/console.html#_2017-06-22_12_57_26_768655

  image api error:

  Jun 22 12:57:25.681138 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.api.v2.image_data [None req-e3efd3d9-5a51-4d12-8ead-c1a5a6577986 c_rally_68e89d0d_oNA8hfKg c_rally_68e89d0d_DJWxs7if] Failed to upload image data due to internal error: IOError: timeout during read(8192) on wsgi.input
  Jun 22 12:57:25.839943 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi [None req-e3efd3d9-5a51-4d12-8ead-c1a5a6577986 c_rally_68e89d0d_oNA8hfKg c_rally_68e89d0d_DJWxs7if] Caught error: timeout during read(8192) on wsgi.input: IOError: timeout during read(8192) on wsgi.input
  Jun 22 12:57:25.840286 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi Traceback (most recent call last):
  Jun 22 12:57:25.840550 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/wsgi.py", line 1157, in __call__
  Jun 22 12:57:25.840747 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     request, **action_args)
  Jun 22 12:57:25.840968 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/wsgi.py", line 1196, in dispatch
  Jun 22 12:57:25.841233 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     return method(*args, **kwargs)
  Jun 22 12:57:25.841498 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/utils.py", line 363, in wrapped
  Jun 22 12:57:25.842109 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
  Jun 22 12:57:25.842305 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/api/v2/image_data.py", line 269, in upload
  Jun 22 12:57:25.842475 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self._restore(image_repo, image)
  Jun 22 12:57:25.842646 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 22 12:57:25.842827 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self.force_reraise()
  Jun 22 12:57:25.843352 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 22 12:57:25.843536 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
  Jun 22 12:57:25.843742 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/api/v2/image_data.py", line 134, in upload
  Jun 22 12:57:25.843929 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     image.set_data(data, size)
  Jun 22 12:57:25.844114 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/domain/proxy.py", line 195, in set_data
  Jun 22 12:57:25.844438 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self.base.set_data(data, size)
  Jun 22 12:57:25.844613 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/notifier.py", line 480, in set_data
  Jun 22 12:57:25.845123 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
  Jun 22 12:57:25.845293 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  Jun 22 12:57:25.845454 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self.force_reraise()
  Jun 22 12:57:25.845615 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Jun 22 12:57:25.845932 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     six.reraise(self.type_, self.value, self.tb)
  Jun 22 12:57:25.846097 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/notifier.py", line 427, in set_data
  Jun 22 12:57:25.846330 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self.repo.set_data(data, size)
  Jun 22 12:57:25.846507 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/api/policy.py", line 192, in set_data
  Jun 22 12:57:25.846693 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
  Jun 22 12:57:25.846870 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/quota/__init__.py", line 304, in set_data
  Jun 22 12:57:25.847046 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self.image.set_data(data, size=size)
  Jun 22 12:57:25.847226 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/location.py", line 439, in set_data
  Jun 22 12:57:25.847393 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     verifier=verifier)
  Jun 22 12:57:25.847559 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 453, in add_to_backend
  Jun 22 12:57:25.847910 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     verifier)
  Jun 22 12:57:25.849074 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 426, in store_add_to_backend
  Jun 22 12:57:25.849255 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     verifier=verifier)
  Jun 22 12:57:25.849418 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py", line 225, in op_checker
  Jun 22 12:57:25.849613 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
  Jun 22 12:57:25.849801 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py", line 873, in add
  Jun 22 12:57:25.849996 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     image_file, content_length=image_size)
  Jun 22 12:57:25.850168 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1824, in put_object
  Jun 22 12:57:25.850354 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     response_dict=response_dict)
  Jun 22 12:57:25.850689 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1673, in _retry
  Jun 22 12:57:25.850881 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     service_token=self.service_token, **kwargs)
  Jun 22 12:57:25.851049 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 1302, in put_object
  Jun 22 12:57:25.851583 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     conn.putrequest(path, headers=headers, data=data)
  Jun 22 12:57:25.851791 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 446, in putrequest
  Jun 22 12:57:25.851971 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     return self.request('PUT', full_path, data, headers, files)
  Jun 22 12:57:25.852156 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 436, in request
  Jun 22 12:57:25.852334 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     files=files, **self.requests_args)
  Jun 22 12:57:25.852612 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/swiftclient/client.py", line 419, in _request
  Jun 22 12:57:25.852802 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     return self.request_session.request(*arg, **kwarg)
  Jun 22 12:57:25.852991 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 513, in request
  Jun 22 12:57:25.853176 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     resp = self.send(prep, **send_kwargs)
  Jun 22 12:57:25.854217 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 623, in send
  Jun 22 12:57:25.854381 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     r = adapter.send(request, **kwargs)
  Jun 22 12:57:25.854540 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/requests/adapters.py", line 440, in send
  Jun 22 12:57:25.854702 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     timeout=timeout
  Jun 22 12:57:25.855722 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
  Jun 22 12:57:25.855889 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     chunked=chunked)
  Jun 22 12:57:25.856043 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py", line 356, in _make_request
  Jun 22 12:57:25.856204 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     conn.request(method, url, **httplib_request_kw)
  Jun 22 12:57:25.856377 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/lib/python2.7/httplib.py", line 1057, in request
  Jun 22 12:57:25.856524 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self._send_request(method, url, body, headers)
  Jun 22 12:57:25.856684 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/lib/python2.7/httplib.py", line 1097, in _send_request
  Jun 22 12:57:25.856846 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self.endheaders(body)
  Jun 22 12:57:25.857006 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/lib/python2.7/httplib.py", line 1053, in endheaders
  Jun 22 12:57:25.857161 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self._send_output(message_body)
  Jun 22 12:57:25.857331 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/lib/python2.7/httplib.py", line 901, in _send_output
  Jun 22 12:57:25.857500 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     self.send(message_body)
  Jun 22 12:57:25.857864 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/lib/python2.7/httplib.py", line 871, in send
  Jun 22 12:57:25.858038 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     datablock = data.read(blocksize)
  Jun 22 12:57:25.858192 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/swiftclient/utils.py", line 296, in read
  Jun 22 12:57:25.858354 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     chunk = self._readable.read(to_read)
  Jun 22 12:57:25.858516 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/utils.py", line 241, in read
  Jun 22 12:57:25.858677 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     result = self.data.read(i)
  Jun 22 12:57:25.858889 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/opt/stack/new/glance/glance/common/utils.py", line 121, in readfn
  Jun 22 12:57:25.859056 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     result = fd.read(*args)
  Jun 22 12:57:25.859226 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1540, in readinto
  Jun 22 12:57:25.859399 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi     data = self.file.read(sz0)
  Jun 22 12:57:25.859551 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi IOError: timeout during read(8192) on wsgi.input
  Jun 22 12:57:25.859730 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: ERROR glance.common.wsgi 
  Jun 22 12:57:25.859895 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: [pid: 20282|app: 0|req: 163/326] 127.0.0.1 () {46 vars in 954 bytes} [Thu Jun 22 12:57:05 2017] PUT /v2/images/ade28c02-f4b2-4153-a3ae-e0d205ba78da/file => generated 228 bytes in 20011 msecs (HTTP/1.1 500) 4 headers in 184 bytes (4 switches on core 0)
  Jun 22 12:57:25.860063 ubuntu-xenial-infracloud-vanilla-9469054 devstack@g-api.service[20280]: [uwsgi-http key: 127.0.0.1:60999 client_addr: 127.0.0.1 client_port: 55446] hr_instance_write(): Connection reset by peer [plugins/http/http.c line 521]

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1700881/+subscriptions
References

[Bug 1700881] [NEW] Not found image after uploading image via volumes.upload_to_image
From: chenhb, 2017-06-28