yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #10570
[Bug 1276142] Re: Physical image can not be deleted again if deletion is failed with OSError
** Changed in: glance
Status: Fix Committed => Fix Released
** Changed in: glance
Milestone: None => icehouse-3
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1276142
Title:
Physical image can not be deleted again if deletion is failed with
OSError
Status in OpenStack Image Registry and Delivery Service (Glance):
Fix Released
Bug description:
If the deletion of physical image failed with OSError for some reason,
it is not possible to delete the image using image-delete api call.
In the longevity test we have encountered this issue, IMO there is
some issue with OS and it is throwing "Forbidden" exception and
somehow its not deleting that file, so in order to reproduce this
issue please follow the below mentioned steps:
Steps to reproduce:
1. image creation, upload
openstack@opencloud1:~$ glance image-create
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | None |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | None |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 0 |
| status | queued |
| updated_at | 2014-01-29T02:27:07 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ glance image-update 675a82ab-4515-451a-932b-6da7f8bce056 --file images/CorePlus4.7.1.qcow2 --disk-format qcow2 --container-format bare
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | active |
| updated_at | 2014-01-29T02:27:20 |
+------------------+--------------------------------------+
2. write protect the image file
openstack@opencloud1:~$ sudo chattr +i
/opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
3. delete the image using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
4. Image is logically deleted, but physically remaining
openstack@opencloud1:~$ glance image-show 675a82ab-4515-451a-932b-
6da7f8bce056
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | True |
| deleted_at | 2014-01-29T02:30:49 |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | deleted |
| updated_at | 2014-01-29T02:30:49 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
5. Remove the write-protect from image file
openstack@opencloud1:~$ sudo chattr -i
/opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
6. Try to delete the image again using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
7. Image file will not get deleted
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
This issue is reproducible only in v1 version. In case of v2 version,
it doesn't delete the meta data associated with the image before
actually deleting it from the backend store.
In glance, if you enable delayed_delete = True and run glance-scrubber
service, glance api will put the image in the queue when the image is
deleted by the user and this image will be deleted asynchronously. If
it encounters the above reported issue while deleting the image in the
glance-scrubber, then this service will simply log error message and
keep on retrying deleting the image until it is deleted finally.
Possible solution:
1. When delayed_delete is enabled.
delete_image_metadata method should be moved to the scrubber and called immediately after setting the image status from "pending_delete" to "deleted".
2. When delayed_delete is not enabled.
delete_image_metadata method should be called after deleting the actual image from the backend.
i.e. after upload_utils.initiate_deletion(req, image['location'], id, CONF.delayed_delete) is called
Also the status of the image should be set back to the previous state in the forbidden exception block.
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1276142/+subscriptions
References
