yahoo-eng-team team mailing list archive

[OpenStack Infra <1885003@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/737867
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=22d8f1fcbf4325d79ac18adfe87198c785f1df52
Submitter: Zuul
Branch:    master

commit 22d8f1fcbf4325d79ac18adfe87198c785f1df52
Author: Abhishek Kekane <akekane@xxxxxxxxxx>
Date:   Wed Jun 24 19:44:54 2020 +0000

    Fix: Interrupted copy-image leaking data on subsequent operation
    
    If copying existing image in other stores fails while staging the data to
    staging directory due to power, network or any other reason. Then
    subsequent try may lead to data leaks in stores.
    
    To fix this, added check of the actual image size with the size
    of image file present in the staging area. If it does not match
    then delete the image file from staging area so that the entire
    image will be staged again.
    
    Change-Id: I44bfefb6eee421e18e5e95a0dafaef0ea4e170da
    Closes-Bug: #1885003


** Changed in: glance
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1885003

Title:
  Interrupted copy-image may break a subsequent operation

Status in Glance:
  Fix Released
Status in Glance ussuri series:
  In Progress
Status in Glance victoria series:
  Fix Released

Bug description:
  Consider this scenario:

  1. Upload image to glance to one store, everything is good
  2. Start an image_import(method='copy-image') to copy the image to another store
  3. Power failure, network failure, or `killall -9 glance-api`
  4. After the failure, re-request the copy-to-store

  At this point, one of two cases will happen (we think) depending on
  the copy request:

  5a. If all_stores_must_succeed=False, then we will see the partial staging residue, try to copy it to the store
  6a. After we copy what was in the staging area to the new store, we will compare the size to that of the actual image, see that it is wrong and fail the operation
  7a. The residue in the staging area will be deleted, but the storage on the backend will neither be updated in locations nor deleted, which is a LEAK (bad).
  8a. The user could retry and it should succeed this time because the staging residue is gone, but the storage was leaked in the above step.

  the other option is:

  5b. If all_stores_must_succeed=True, then we will see the partial staging residue, try to copy it to the store
  6b. After we copy what was in the staging area to the new store and compare the size, we will fail the operation
  7b. We will not delete the residue from the staging dir, but _will_ delete the backend storage, avoiding the leak.
  8b. The user will retry, which will repeat the same and fail again, over and over.

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1885003/+subscriptions