← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #87519

[Bug 1948706] [NEW] Glance cannot remove image if Nova boots instance from image with incorrect signature.

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
Nova is configured to verify glance images:
[glance]
verify_glance_signatures=true

Glance backend is Ceph.


Steps to reproduce
==================
1. create glance image with proper signature
2. update glance image with incorrect signature
3. try to boot instance from the glance image with incorrect signature.
Boot fails because Nova checks signature and verification fails. 
It's correct behavior.

barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_boot_failure[compute,id-74f022d6-a6ef-4458-96b7-541deadacf99,image,smoke]
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/services/image/v2/images_client.py", line 103, in delete_image
    resp, _ = self.delete(url)

      File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 330, in delete
    return self.request('DELETE', url, extra_headers, headers, body)

      File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 710, in request
    self._error_checker(resp, resp_body)  

      File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 831, in _error_checker
    raise exceptions.Conflict(resp_body, resp=resp)

    tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'message': 'Image c321f6be-a4d3-42d2-bc3f-f0ea913b83b7 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.<br /><br />\n\n\n', 'code': '409 Conflict', 'title': 'Conflict'}

4. Delete the glance image right after failed instance boot.

Expected result
===============
Glance image was deleted successfully. 

Actual result
=============
Glance cannot be deleted.
In Glance backend we see that there are watchers that protect glance image from deletion:

# rbd  rm --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
2021-10-15T13:25:03.862+0000 7f36b98c8700 -1 librbd::image::PreRemoveRequest: 0x562785d77a50 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

# rbd  status  --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
Watchers:
        watcher=10.10.0.89:0/729945307 client.374098 cookie=140684808072160

The behavior is reproduced by tempest test:
https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/test_image_signing.py#L67

Environment
===========
1. Openstack version: Victoria
2. Hypervisor: KVM + libvirt
3. Glance storage: Ceph, Nova storage: local.
4. Networking: Neutron with OVS

** Affects: nova
     Importance: Undecided
     Assignee: Mitya Eremeev (mitos)
         Status: In Progress

** Changed in: nova
     Assignee: (unassigned) => Mitya Eremeev (mitos)

** Changed in: nova
     Assignee: Mitya Eremeev (mitos) => (unassigned)

** Changed in: nova
     Assignee: (unassigned) => Mitya Eremeev (mitos)

** Changed in: nova
       Status: New => In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1948706

Title:
  Glance cannot remove image if Nova boots instance from image with
  incorrect signature.

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========
  Nova is configured to verify glance images:
  [glance]
  verify_glance_signatures=true

  Glance backend is Ceph.

  
  Steps to reproduce
  ==================
  1. create glance image with proper signature
  2. update glance image with incorrect signature
  3. try to boot instance from the glance image with incorrect signature.
  Boot fails because Nova checks signature and verification fails. 
  It's correct behavior.

  barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_boot_failure[compute,id-74f022d6-a6ef-4458-96b7-541deadacf99,image,smoke]
  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  Captured traceback:
  ~~~~~~~~~~~~~~~~~~~
      Traceback (most recent call last):

        File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/services/image/v2/images_client.py", line 103, in delete_image
      resp, _ = self.delete(url)

        File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 330, in delete
      return self.request('DELETE', url, extra_headers, headers, body)

        File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 710, in request
      self._error_checker(resp, resp_body)  

        File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 831, in _error_checker
      raise exceptions.Conflict(resp_body, resp=resp)

      tempest.lib.exceptions.Conflict: Conflict with state of target resource
  Details: {'message': 'Image c321f6be-a4d3-42d2-bc3f-f0ea913b83b7 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.<br /><br />\n\n\n', 'code': '409 Conflict', 'title': 'Conflict'}

  4. Delete the glance image right after failed instance boot.

  Expected result
  ===============
  Glance image was deleted successfully. 

  Actual result
  =============
  Glance cannot be deleted.
  In Glance backend we see that there are watchers that protect glance image from deletion:

  # rbd  rm --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
  2021-10-15T13:25:03.862+0000 7f36b98c8700 -1 librbd::image::PreRemoveRequest: 0x562785d77a50 check_image_watchers: image has watchers - not removing
  Removing image: 0% complete...failed.
  rbd: error: image still has watchers
  This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

  # rbd  status  --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
  Watchers:
          watcher=10.10.0.89:0/729945307 client.374098 cookie=140684808072160

  The behavior is reproduced by tempest test:
  https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/test_image_signing.py#L67

  Environment
  ===========
  1. Openstack version: Victoria
  2. Hypervisor: KVM + libvirt
  3. Glance storage: Ceph, Nova storage: local.
  4. Networking: Neutron with OVS

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1948706/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next