yahoo-eng-team team mailing list archive

[OpenStack Infra <1948706@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/815347
Committed: https://opendev.org/openstack/nova/commit/43bca185fe2d00bb70d7486fa6c6a0b9eda1fc17
Submitter: "Zuul (22348)"
Branch:    master

commit 43bca185fe2d00bb70d7486fa6c6a0b9eda1fc17
Author: Mitya_Eremeev <mitossvyaz@xxxxxxx>
Date:   Thu Nov 11 18:52:11 2021 +0300

    Close Glance image if downloading failed.
    
    If downloding of Glance image failed we should
    close iterator of image body.
    Otherwise Glance is unable to delete the image.
    
    Change-Id: I193df2fcbf2588c10be953eb4e9eef4609b6286f
    Closes-Bug: 1948706


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1948706

Title:
  Glance cannot remove image if Nova boots instance from image with
  incorrect signature.

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  Nova is configured to verify glance images:
  [glance]
  verify_glance_signatures=true

  Glance backend is Ceph.

  
  Steps to reproduce
  ==================
  1. create glance image with proper signature
  2. update glance image with incorrect signature
  3. try to boot instance from the glance image with incorrect signature.
  Boot fails because Nova checks signature and verification fails. 
  It's correct behavior.

  barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_boot_failure[compute,id-74f022d6-a6ef-4458-96b7-541deadacf99,image,smoke]
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  Captured traceback:
  ~~~~~~~~~~~~~~~~~~~
      Traceback (most recent call last):

        File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/services/image/v2/images_client.py", line 103, in delete_image
      resp, _ = self.delete(url)

        File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 330, in delete
      return self.request('DELETE', url, extra_headers, headers, body)

        File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 710, in request
      self._error_checker(resp, resp_body)  

        File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 831, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)

      tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'message': 'Image c321f6be-a4d3-42d2-bc3f-f0ea913b83b7 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.<br /><br />\n\n\n', 'code': '409 Conflict', 'title': 'Conflict'}

  4. Delete the glance image right after failed instance boot.

  Expected result
  ===============
  Glance image was deleted successfully. 

  Actual result
  =============
  Glance cannot be deleted.
  In Glance backend we see that there are watchers that protect glance image from deletion:

  # rbd  rm --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
  2021-10-15T13:25:03.862+0000 7f36b98c8700 -1 librbd::image::PreRemoveRequest: 0x562785d77a50 check_image_watchers: image has watchers - not removing
  Removing image: 0% complete...failed.
  rbd: error: image still has watchers
  This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

  # rbd  status  --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
  Watchers:
          watcher=10.10.0.89:0/729945307 client.374098 cookie=140684808072160

  The behavior is reproduced by tempest test:
  https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/test_image_signing.py#L67

  Environment
  ===========
  1. Openstack version: Victoria
  2. Hypervisor: KVM + libvirt
  3. Glance storage: Ceph, Nova storage: local.
  4. Networking: Neutron with OVS

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1948706/+subscriptions