yahoo-eng-team team mailing list archive

[Dan Smith <1913625@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

In various situations, glance will leak (potentially very large)
temporary files in the staging store.

One example is doing a web-download import, where glance initially
downloads the image to its staging store. If the worker doing that
activity crashes, loses power, etc, the user may delete the image and
try again on another worker. When the crashed worker resumes, the
staging data will remain but nothing will ever clean it up.

Another example would be a misconfigured glance that uses local staging
directories, but glance-direct is used, where the user stages data, and
then deletes the image from another worker.

Even in a situation where shared staging is properly configured, a
failure to access the staging location during the delete call will
result in the image being deleted, but the staging file not being
purged.

IMHO, glance workers should clean their staging directories at startup,
purging any data that is attributable to a previous image having been
deleted.

Another option is to add a store location for each staged image, and
make sure the scrubber can clean those things from the staging directory
periodically (this requires also running the scrubber on each node,
which may not be common practice currently).

** Affects: glance
     Importance: Undecided
         Status: Invalid

** Changed in: glance
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1913625

Title:
  Glance will leak staging data

Status in Glance:
  Invalid

Bug description:
  In various situations, glance will leak (potentially very large)
  temporary files in the staging store.

  One example is doing a web-download import, where glance initially
  downloads the image to its staging store. If the worker doing that
  activity crashes, loses power, etc, the user may delete the image and
  try again on another worker. When the crashed worker resumes, the
  staging data will remain but nothing will ever clean it up.

  Another example would be a misconfigured glance that uses local
  staging directories, but glance-direct is used, where the user stages
  data, and then deletes the image from another worker.

  Even in a situation where shared staging is properly configured, a
  failure to access the staging location during the delete call will
  result in the image being deleted, but the staging file not being
  purged.

  IMHO, glance workers should clean their staging directories at
  startup, purging any data that is attributable to a previous image
  having been deleted.

  Another option is to add a store location for each staged image, and
  make sure the scrubber can clean those things from the staging
  directory periodically (this requires also running the scrubber on
  each node, which may not be common practice currently).

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1913625/+subscriptions