yahoo-eng-team team mailing list archive

[OpenStack Infra <1950657@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/818503
Committed: https://opendev.org/openstack/nova/commit/ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Submitter: "Zuul (22348)"
Branch:    master

commit ce493273b9404530dfa8ecfe3eaa3d6c81a20e39
Author: sdmitriev1 <sdmitriev1@xxxxxxxxx>
Date:   Thu Nov 18 22:05:05 2021 -0500

    Retry image download if it's corrupted
    
    Adding IOError in list of catching exceptions in order to
    fix behavior when nova-compute wouldn't retry image download
    when got "Corrupt image download" error from glanceclient
    and had num_retries config option set.
    
    Closes-Bug: #1950657
    Change-Id: Iae4fd0579f71d3ba6793dbdb037275352d7e57b0


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1950657

Title:
  Nova-compute wouldn't retry image download when gets "Corrupt image
  download" error

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Nova-compute wouldn't retry image download when gets "Corrupt image
  download" error from glanceclient.

  There is a configuration option num_retries (equal 3 by default) in
  glance section of nova-config file, so nova-compute supposed to retry
  image download if it failed, but it wouldn't work in case of next
  exception:

  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 375, in download
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]     for chunk in image_chunks:
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]   File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 519, in __iter__
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]     for chunk in self.iterable:
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]   File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 469, in serious_integrity_iter
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]     (computed, hash_value))
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] IOError: [Errno 32] Corrupt image download. Hash was 12e58a8b858a560ba89a035c24c3453bb19a294b1cc59088ff3d9f414053c7cdd84b323510dc8c30eb560a813cd670caa6ef9f56e12ae1213f12680aea039f53 expected a37eacb7894f4e76c7511b6f5862b246776e3a2ccfdd195894170866650a63b67353c2a53c1898e4b079e280d43f09f27ced6a057d16cc93018b71ac13c26bd7
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] 

  
  It wouldn't work because IOError exception is not in retry_excs list:
  https://github.com/openstack/nova/blob/master/nova/image/glance.py#L179

          retry_excs = (glanceclient.exc.ServiceUnavailable,
                  glanceclient.exc.InvalidEndpoint,
                  glanceclient.exc.CommunicationError)

  so try-except block doesn't catch and download retry never happens

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1950657/+subscriptions