yahoo-eng-team team mailing list archive

[melanie witt <1950657@xxxxxxxxxxxxxxxxxx>]

** Also affects: nova/victoria
   Importance: Undecided
       Status: New

** Also affects: nova/wallaby
   Importance: Undecided
       Status: New

** Also affects: nova/xena
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1950657

Title:
  Nova-compute wouldn't retry image download when gets "Corrupt image
  download" error

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) victoria series:
  New
Status in OpenStack Compute (nova) wallaby series:
  New
Status in OpenStack Compute (nova) xena series:
  New

Bug description:
  Nova-compute wouldn't retry image download when gets "Corrupt image
  download" error from glanceclient.

  There is a configuration option num_retries (equal 3 by default) in
  glance section of nova-config file, so nova-compute supposed to retry
  image download if it failed, but it wouldn't work in case of next
  exception:

  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]   File "/usr/lib/python2.7/site-packages/nova/image/glance.py", line 375, in download
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]     for chunk in image_chunks:
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]   File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 519, in __iter__
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]     for chunk in self.iterable:
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]   File "/usr/lib/python2.7/site-packages/glanceclient/common/utils.py", line 469, in serious_integrity_iter
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89]     (computed, hash_value))
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] IOError: [Errno 32] Corrupt image download. Hash was 12e58a8b858a560ba89a035c24c3453bb19a294b1cc59088ff3d9f414053c7cdd84b323510dc8c30eb560a813cd670caa6ef9f56e12ae1213f12680aea039f53 expected a37eacb7894f4e76c7511b6f5862b246776e3a2ccfdd195894170866650a63b67353c2a53c1898e4b079e280d43f09f27ced6a057d16cc93018b71ac13c26bd7
  2021-11-02 10:42:34.192 6 ERROR nova.compute.manager [instance: ec0f0736-a5cd-48dc-b2f9-851629604a89] 

  
  It wouldn't work because IOError exception is not in retry_excs list:
  https://github.com/openstack/nova/blob/master/nova/image/glance.py#L179

          retry_excs = (glanceclient.exc.ServiceUnavailable,
                  glanceclient.exc.InvalidEndpoint,
                  glanceclient.exc.CommunicationError)

  so try-except block doesn't catch and download retry never happens

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1950657/+subscriptions