yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #87636
[Bug 1950657] [NEW] Nova-compute wouldn't retry image download when gets "Corrupt image download" error
Public bug reported:
Nova-compute wouldn't retry image download when gets "Corrupt image
download" error from glanceclient.
There is a configuration option num_retries (equal 3 by default) in
glance section of nova-config file, so nova-compute supposed to retry
image download if it failed, but it wouldn't work in case of next
exception:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 375, in download
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in image_chunks:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 519, in __iter__
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in self.iterable:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 469, in serious_integrity_iter
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] (computed, hash_value))
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] IOError: [Errno 32] Corrupt image download. Hash was 12e58a8b858a560ba89a035c24c3453bb19a294b1cc59088ff3d9f414053c7cdd84b323510dc8c30eb560a813cd670caa6ef9f56e12ae1213f12680aea039f53 expected a37eacb7894f4e76c7511b6f5862b246776e3a2ccfdd195894170866650a63b67353c2a53c1898e4b079e280d43f09f27ced6a057d16cc93018b71ac13c26bd7
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]
It wouldn't work because IOError exception is not in retry_excs list:
https://github.com/openstack/nova/blob/master/nova/image/glance.py#L179
retry_excs = (glanceclient.exc.ServiceUnavailable,
glanceclient.exc.InvalidEndpoint,
glanceclient.exc.CommunicationError)
so try-except block doesn't catch and download retry never happens
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1950657
Title:
Nova-compute wouldn't retry image download when gets "Corrupt image
download" error
Status in OpenStack Compute (nova):
New
Bug description:
Nova-compute wouldn't retry image download when gets "Corrupt image
download" error from glanceclient.
There is a configuration option num_retries (equal 3 by default) in
glance section of nova-config file, so nova-compute supposed to retry
image download if it failed, but it wouldn't work in case of next
exception:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 375, in download
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in image_chunks:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 519, in __iter__
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] for chunk in self.iterable:
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 469, in serious_integrity_iter
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] (computed, hash_value))
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] IOError: [Errno 32] Corrupt image download. Hash was 12e58a8b858a560ba89a035c24c3453bb19a294b1cc59088ff3d9f414053c7cdd84b323510dc8c30eb560a813cd670caa6ef9f56e12ae1213f12680aea039f53 expected a37eacb7894f4e76c7511b6f5862b246776e3a2ccfdd195894170866650a63b67353c2a53c1898e4b079e280d43f09f27ced6a057d16cc93018b71ac13c26bd7
2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]
It wouldn't work because IOError exception is not in retry_excs list:
https://github.com/openstack/nova/blob/master/nova/image/glance.py#L179
retry_excs = (glanceclient.exc.ServiceUnavailable,
glanceclient.exc.InvalidEndpoint,
glanceclient.exc.CommunicationError)
so try-except block doesn't catch and download retry never happens
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1950657/+subscriptions
Follow ups