yahoo-eng-team team mailing list archive

[nikhil komawar <nik.komawar@xxxxxxxxx>]

1. I agree, the image deletion operation should be atomic.

2. Image data left behind means there is a potential risk of filling up
storage quota and resulting into a DoS; be mindful that it's a denial of
service but NOT a exploit as it is dependent on the operators' failure
scenarios of g-api <-> reg communication.

3. The original description has information related to failure scenarios
for only v1. So, a check is needed for the v2 as applicable.

** Description changed:

  [0] shows a sample image that was left in an inconsistent state when a
  call to registry failed during image deletion.
  
- Glance API makes two registry calls when deleting an image.
+ Glance v1 API makes two registry calls when deleting an image.
  The first call [1] is made to to set the status of an image to deleted/pending_delete.
  And, the other [2], to delete the rest of the metadata, which sets 'deleted_at' and 'deleted' fields in the db.
  
  If the first call fails, the image deletion request fails and the image is left intact in it's previous status.
  However, if the first call succeeds and the second one fails, the image is left in an inconsistent status where it's status is set to pending_delete/deleted but it's 'deleted_at' and 'deleted' fields are not set.
  
  If delayed delete is turned on, these images are never collected by the scrubber as they won't appear as deleted images because their deleted field is not set. So, these images will continue to occupy storage in the backend.
  Also, further attempts at deleting these images will fail with a 404 because the status is already set to pending_delete/deleted.
  
  [0] http://paste.openstack.org/show/477577/
  [1]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1115-L1116
  [2]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1132

** Changed in: glance
       Status: New => Triaged

** Changed in: glance
   Importance: Undecided => Critical

** Changed in: glance
    Milestone: None => mitaka-1

** Changed in: glance
     Assignee: (unassigned) => Hemanth Makkapati (hemanth-makkapati)

** Also affects: glance/kilo
   Importance: Undecided
       Status: New

** Also affects: glance/juno
   Importance: Undecided
       Status: New

** Also affects: glance/liberty
   Importance: Undecided
       Status: New

** Information type changed from Public to Public Security

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1511061

Title:
  Images in inconsistent state when calls to registry fail during image
  deletion

Status in Glance:
  Triaged
Status in Glance juno series:
  New
Status in Glance kilo series:
  New
Status in Glance liberty series:
  New

Bug description:
  [0] shows a sample image that was left in an inconsistent state when a
  call to registry failed during image deletion.

  Glance v1 API makes two registry calls when deleting an image.
  The first call [1] is made to to set the status of an image to deleted/pending_delete.
  And, the other [2], to delete the rest of the metadata, which sets 'deleted_at' and 'deleted' fields in the db.

  If the first call fails, the image deletion request fails and the image is left intact in it's previous status.
  However, if the first call succeeds and the second one fails, the image is left in an inconsistent status where it's status is set to pending_delete/deleted but it's 'deleted_at' and 'deleted' fields are not set.

  If delayed delete is turned on, these images are never collected by the scrubber as they won't appear as deleted images because their deleted field is not set. So, these images will continue to occupy storage in the backend.
  Also, further attempts at deleting these images will fail with a 404 because the status is already set to pending_delete/deleted.

  [0] http://paste.openstack.org/show/477577/
  [1]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1115-L1116
  [2]: https://github.com/openstack/glance/blob/master/glance/api/v1/images.py#L1132

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1511061/+subscriptions