yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #91590
[Bug 2012898] [NEW] glance download interrupted with glance cache enabled and checksum verification failed
Public bug reported:
Description
===========
The process of image download in glance with caching option push the image to the client and in the same time copy the image in the local cache.
At the end of the download, cache do a checksum check before writing the image to the cache. If the checksum is wrong, we raise exception in this part of code.
This exception stop the streaming of the image for the customer
Steps to reproduce
==================
Case with cache enabled:
* Enable glance cache, to do this, add in [paste_deploy] section:
flavor=keystone+cachemanagement
* create a new image with checksum
$ openstack image create cirros-image-test-download --file cirros-image
* change the checksum in openstack database (to reproduce each time the issue)
$ update images set checksum="a3102e9586409fd6da9885252fd44c66" where id="c35e644a-21cd-4852-9dab-190ba5032d53"
* download the image
openstack image save c35e644a-21cd-4852-9dab-190ba5032d53 --file cirros-image-test
Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
Case with cache disabled
Without the cache, in [paste_deploy] section:
flavor=keystone
* download the image
openstack image save c35e644a-21cd-4852-9dab-190ba5032d53 --file cirros-image-test
Unable to download image: InvalidResponse
Expected result
===============
We want the same result with cache management enable or disable
Note that the InvalidResponse in openstack client is a good answer, because the checksum is wrong and the client is checking the checksum
Actual result
=============
We have stack trace in glance about Checksum verification failed. This error broke the download of the image for the client and generate error:
Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
Environment
===========
We used glance in Yoga version and two configuration for the test are:
1. Enable glance cache with "flavor=keystone+cachemanagement" in [paste_deploy] section
2. Disable glance cache with "flavor=keystone" in [paste_deploy] section
Logs & Configs
==============
Log in glance with cache enabled:
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache [req-ca460553-5b44-4a62-badd-3ad942e56710 b1d4870dfa644c7fb0d515bd982d9c91 bf02b386ec434f06b66036313638e66a - default default] Checksum verification failed. Aborted caching of image 'c35e644a-21cd-4852-9dab-190ba5032d53'.: glance.common.exception.GlanceException: Checksum verification failed. Aborted caching of image 'c35e644a-21cd-4852-9dab-190ba5032d53'.
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache Traceback (most recent call last):
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache File "/opt/openstack/glance/lib/python3.7/site-packages/glance/image_cache/__init__.py", line 364, in cache_tee_iter
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache raise exception.GlanceException(msg)
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache glance.common.exception.GlanceException: Checksum verification failed. Aborted caching of image 'c35e644a-21cd-4852-9dab-190ba5032d53'.
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache
2023-03-24 15:35:29.054 26671 INFO eventlet.wsgi.server [req-ca460553-5b44-4a62-badd-3ad942e56710 b1d4870dfa644c7fb0d515bd982d9c91 bf02b386ec434f06b66036313638e66a - default default] Traceback (most recent call last):
File "/opt/openstack/glance/lib/python3.7/site-packages/eventlet/wsgi.py", line 587, in handle_one_response
for data in result:
File "/opt/openstack/glance/lib/python3.7/site-packages/glance/image_cache/__init__.py", line 370, in cache_tee_iter
LOG.exception(encodeutils.exception_to_unicode(e))
File "/opt/openstack/glance/lib/python3.7/site-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/opt/openstack/glance/lib/python3.7/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/opt/openstack/glance/lib/python3.7/site-packages/glance/image_cache/__init__.py", line 364, in cache_tee_iter
raise exception.GlanceException(msg)
glance.common.exception.GlanceException: Checksum verification failed. Aborted caching of image 'c35e644a-21cd-4852-9dab-190ba5032d53'.
2023-03-24 15:35:29.054 26671 INFO eventlet.wsgi.server [req-ca460553-5b44-4a62-badd-3ad942e56710 b1d4870dfa644c7fb0d515bd982d9c91 bf02b386ec434f06b66036313638e66a - default default] 148.113.141.123,10.109.128.227 - - [24/Mar/2023 15:35:29] "GET /v2/images/c35e644a-21cd-4852-9dab-190ba5032d53/file HTTP/1.1" 200 31789562 2.498400
Log openstack client:
https://image.compute.dev.cloud.ovh.net:9090 "GET /v2/images/c35e644a-21cd-4852-9dab-190ba5032d53/file HTTP/1.1" 200 None
RESP: [200] Content-Md5: a3102e9586409fd6da9885252fd44c64 Content-Type: application/octet-stream Date: Mon, 27 Mar 2023 07:58:42 GMT Strict-Transport-Security: max-age=15768000 Transfer-Encoding: chunked X-IPLB-Instance: 32288 X-IPLB-Request-ID: 94718D7B:C3A0_894A7DD0:2382_64214CB1_0F39:1C222 X-Openstack-Request-Id: req-0935da00-5598-4a0e-b935-9d85e1874297
RESP BODY: Omitted, Content-Type is set to application/octet-stream. Only text/plain, application/json responses have their bodies logged.
GET call to image for https://image.compute.dev.cloud.ovh.net:9090/v2/images/c35e644a-21cd-4852-9dab-190ba5032d53/file used request id req-0935da00-5598-4a0e-b935-9d85e1874297
Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 700, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 441, in _error_catcher
yield
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 767, in read_chunked
self._update_chunk_length()
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 704, in _update_chunk_length
raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/requests/models.py", line 760, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 575, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 796, in read_chunked
self._original_response.close()
File "/opt/venvs/ovh-openstackclient/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 458, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
chunk_size=chunk_size):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/requests/models.py", line 763, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
result = cmd.run(parsed_args)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
return super(Command, self).run(parsed_args)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/cliff/command.py", line 186, in run
return_code = self.take_action(parsed_args) or 0
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 846, in take_action
image_client.download_image(image.id, stream=True, output=output_file)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
self, stream=stream, output=output, chunk_size=chunk_size)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
"Unable to download image: %s" % e)
openstack.exceptions.SDKException: Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
clean_up SaveImage: Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
** Affects: glance
Importance: Undecided
Assignee: Pierre Libeau (pierre-libeau)
Status: New
** Changed in: glance
Assignee: (unassigned) => Pierre Libeau (pierre-libeau)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/2012898
Title:
glance download interrupted with glance cache enabled and checksum
verification failed
Status in Glance:
New
Bug description:
Description
===========
The process of image download in glance with caching option push the image to the client and in the same time copy the image in the local cache.
At the end of the download, cache do a checksum check before writing the image to the cache. If the checksum is wrong, we raise exception in this part of code.
This exception stop the streaming of the image for the customer
Steps to reproduce
==================
Case with cache enabled:
* Enable glance cache, to do this, add in [paste_deploy] section:
flavor=keystone+cachemanagement
* create a new image with checksum
$ openstack image create cirros-image-test-download --file cirros-image
* change the checksum in openstack database (to reproduce each time the issue)
$ update images set checksum="a3102e9586409fd6da9885252fd44c66" where id="c35e644a-21cd-4852-9dab-190ba5032d53"
* download the image
openstack image save c35e644a-21cd-4852-9dab-190ba5032d53 --file cirros-image-test
Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
Case with cache disabled
Without the cache, in [paste_deploy] section:
flavor=keystone
* download the image
openstack image save c35e644a-21cd-4852-9dab-190ba5032d53 --file cirros-image-test
Unable to download image: InvalidResponse
Expected result
===============
We want the same result with cache management enable or disable
Note that the InvalidResponse in openstack client is a good answer, because the checksum is wrong and the client is checking the checksum
Actual result
=============
We have stack trace in glance about Checksum verification failed. This error broke the download of the image for the client and generate error:
Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
Environment
===========
We used glance in Yoga version and two configuration for the test are:
1. Enable glance cache with "flavor=keystone+cachemanagement" in [paste_deploy] section
2. Disable glance cache with "flavor=keystone" in [paste_deploy] section
Logs & Configs
==============
Log in glance with cache enabled:
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache [req-ca460553-5b44-4a62-badd-3ad942e56710 b1d4870dfa644c7fb0d515bd982d9c91 bf02b386ec434f06b66036313638e66a - default default] Checksum verification failed. Aborted caching of image 'c35e644a-21cd-4852-9dab-190ba5032d53'.: glance.common.exception.GlanceException: Checksum verification failed. Aborted caching of image 'c35e644a-21cd-4852-9dab-190ba5032d53'.
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache Traceback (most recent call last):
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache File "/opt/openstack/glance/lib/python3.7/site-packages/glance/image_cache/__init__.py", line 364, in cache_tee_iter
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache raise exception.GlanceException(msg)
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache glance.common.exception.GlanceException: Checksum verification failed. Aborted caching of image 'c35e644a-21cd-4852-9dab-190ba5032d53'.
2023-03-24 15:35:29.052 26671 ERROR glance.image_cache
2023-03-24 15:35:29.054 26671 INFO eventlet.wsgi.server [req-ca460553-5b44-4a62-badd-3ad942e56710 b1d4870dfa644c7fb0d515bd982d9c91 bf02b386ec434f06b66036313638e66a - default default] Traceback (most recent call last):
File "/opt/openstack/glance/lib/python3.7/site-packages/eventlet/wsgi.py", line 587, in handle_one_response
for data in result:
File "/opt/openstack/glance/lib/python3.7/site-packages/glance/image_cache/__init__.py", line 370, in cache_tee_iter
LOG.exception(encodeutils.exception_to_unicode(e))
File "/opt/openstack/glance/lib/python3.7/site-packages/oslo_utils/excutils.py", line 227, in __exit__
self.force_reraise()
File "/opt/openstack/glance/lib/python3.7/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
raise self.value
File "/opt/openstack/glance/lib/python3.7/site-packages/glance/image_cache/__init__.py", line 364, in cache_tee_iter
raise exception.GlanceException(msg)
glance.common.exception.GlanceException: Checksum verification failed. Aborted caching of image 'c35e644a-21cd-4852-9dab-190ba5032d53'.
2023-03-24 15:35:29.054 26671 INFO eventlet.wsgi.server [req-ca460553-5b44-4a62-badd-3ad942e56710 b1d4870dfa644c7fb0d515bd982d9c91 bf02b386ec434f06b66036313638e66a - default default] 148.113.141.123,10.109.128.227 - - [24/Mar/2023 15:35:29] "GET /v2/images/c35e644a-21cd-4852-9dab-190ba5032d53/file HTTP/1.1" 200 31789562 2.498400
Log openstack client:
https://image.compute.dev.cloud.ovh.net:9090 "GET /v2/images/c35e644a-21cd-4852-9dab-190ba5032d53/file HTTP/1.1" 200 None
RESP: [200] Content-Md5: a3102e9586409fd6da9885252fd44c64 Content-Type: application/octet-stream Date: Mon, 27 Mar 2023 07:58:42 GMT Strict-Transport-Security: max-age=15768000 Transfer-Encoding: chunked X-IPLB-Instance: 32288 X-IPLB-Request-ID: 94718D7B:C3A0_894A7DD0:2382_64214CB1_0F39:1C222 X-Openstack-Request-Id: req-0935da00-5598-4a0e-b935-9d85e1874297
RESP BODY: Omitted, Content-Type is set to application/octet-stream. Only text/plain, application/json responses have their bodies logged.
GET call to image for https://image.compute.dev.cloud.ovh.net:9090/v2/images/c35e644a-21cd-4852-9dab-190ba5032d53/file used request id req-0935da00-5598-4a0e-b935-9d85e1874297
Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 700, in _update_chunk_length
self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 441, in _error_catcher
yield
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 767, in read_chunked
self._update_chunk_length()
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 704, in _update_chunk_length
raise InvalidChunkLength(self, line)
urllib3.exceptions.InvalidChunkLength: InvalidChunkLength(got length b'', 0 bytes read)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/requests/models.py", line 760, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 575, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 796, in read_chunked
self._original_response.close()
File "/opt/venvs/ovh-openstackclient/lib/python3.6/contextlib.py", line 99, in __exit__
self.gen.throw(type, value, traceback)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/urllib3/response.py", line 458, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/openstack/image/_download.py", line 57, in download
chunk_size=chunk_size):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/requests/models.py", line 763, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/cliff/app.py", line 407, in run_subcommand
result = cmd.run(parsed_args)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run
return super(Command, self).run(parsed_args)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/cliff/command.py", line 186, in run
return_code = self.take_action(parsed_args) or 0
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/openstackclient/image/v2/image.py", line 846, in take_action
image_client.download_image(image.id, stream=True, output=output_file)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/openstack/image/v2/_proxy.py", line 451, in download_image
self, stream=stream, output=output, chunk_size=chunk_size)
File "/opt/venvs/ovh-openstackclient/lib/python3.6/site-packages/openstack/image/_download.py", line 65, in download
"Unable to download image: %s" % e)
openstack.exceptions.SDKException: Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
clean_up SaveImage: Unable to download image: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read))
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/2012898/+subscriptions
