yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #87519
[Bug 1948706] [NEW] Glance cannot remove image if Nova boots instance from image with incorrect signature.
Public bug reported:
Description
===========
Nova is configured to verify glance images:
[glance]
verify_glance_signatures=true
Glance backend is Ceph.
Steps to reproduce
==================
1. create glance image with proper signature
2. update glance image with incorrect signature
3. try to boot instance from the glance image with incorrect signature.
Boot fails because Nova checks signature and verification fails.
It's correct behavior.
barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_boot_failure[compute,id-74f022d6-a6ef-4458-96b7-541deadacf99,image,smoke]
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/services/image/v2/images_client.py", line 103, in delete_image
resp, _ = self.delete(url)
File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 330, in delete
return self.request('DELETE', url, extra_headers, headers, body)
File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 710, in request
self._error_checker(resp, resp_body)
File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 831, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'message': 'Image c321f6be-a4d3-42d2-bc3f-f0ea913b83b7 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.<br /><br />\n\n\n', 'code': '409 Conflict', 'title': 'Conflict'}
4. Delete the glance image right after failed instance boot.
Expected result
===============
Glance image was deleted successfully.
Actual result
=============
Glance cannot be deleted.
In Glance backend we see that there are watchers that protect glance image from deletion:
# rbd rm --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
2021-10-15T13:25:03.862+0000 7f36b98c8700 -1 librbd::image::PreRemoveRequest: 0x562785d77a50 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
# rbd status --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
Watchers:
watcher=10.10.0.89:0/729945307 client.374098 cookie=140684808072160
The behavior is reproduced by tempest test:
https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/test_image_signing.py#L67
Environment
===========
1. Openstack version: Victoria
2. Hypervisor: KVM + libvirt
3. Glance storage: Ceph, Nova storage: local.
4. Networking: Neutron with OVS
** Affects: nova
Importance: Undecided
Assignee: Mitya Eremeev (mitos)
Status: In Progress
** Changed in: nova
Assignee: (unassigned) => Mitya Eremeev (mitos)
** Changed in: nova
Assignee: Mitya Eremeev (mitos) => (unassigned)
** Changed in: nova
Assignee: (unassigned) => Mitya Eremeev (mitos)
** Changed in: nova
Status: New => In Progress
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1948706
Title:
Glance cannot remove image if Nova boots instance from image with
incorrect signature.
Status in OpenStack Compute (nova):
In Progress
Bug description:
Description
===========
Nova is configured to verify glance images:
[glance]
verify_glance_signatures=true
Glance backend is Ceph.
Steps to reproduce
==================
1. create glance image with proper signature
2. update glance image with incorrect signature
3. try to boot instance from the glance image with incorrect signature.
Boot fails because Nova checks signature and verification fails.
It's correct behavior.
barbican_tempest_plugin.tests.scenario.test_image_signing.ImageSigningTest.test_signed_image_upload_boot_failure[compute,id-74f022d6-a6ef-4458-96b7-541deadacf99,image,smoke]
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Captured traceback:
~~~~~~~~~~~~~~~~~~~
Traceback (most recent call last):
File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/services/image/v2/images_client.py", line 103, in delete_image
resp, _ = self.delete(url)
File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 330, in delete
return self.request('DELETE', url, extra_headers, headers, body)
File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 710, in request
self._error_checker(resp, resp_body)
File "/var/lib/openstack/lib/python3.6/site-packages/tempest/lib/common/rest_client.py", line 831, in _error_checker
raise exceptions.Conflict(resp_body, resp=resp)
tempest.lib.exceptions.Conflict: Conflict with state of target resource
Details: {'message': 'Image c321f6be-a4d3-42d2-bc3f-f0ea913b83b7 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.<br /><br />\n\n\n', 'code': '409 Conflict', 'title': 'Conflict'}
4. Delete the glance image right after failed instance boot.
Expected result
===============
Glance image was deleted successfully.
Actual result
=============
Glance cannot be deleted.
In Glance backend we see that there are watchers that protect glance image from deletion:
# rbd rm --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
2021-10-15T13:25:03.862+0000 7f36b98c8700 -1 librbd::image::PreRemoveRequest: 0x562785d77a50 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
# rbd status --pool images-hdd c321f6be-a4d3-42d2-bc3f-f0ea913b83b7
Watchers:
watcher=10.10.0.89:0/729945307 client.374098 cookie=140684808072160
The behavior is reproduced by tempest test:
https://github.com/openstack/barbican-tempest-plugin/blob/master/barbican_tempest_plugin/tests/scenario/test_image_signing.py#L67
Environment
===========
1. Openstack version: Victoria
2. Hypervisor: KVM + libvirt
3. Glance storage: Ceph, Nova storage: local.
4. Networking: Neutron with OVS
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1948706/+subscriptions
Follow ups