yahoo-eng-team team mailing list archive

[Chris Friesen <chris.friesen@xxxxxxxxxxxxx>]

Public bug reported:

When creating larger glance images (like a 10GB CentOS7 image), if we
run into situation where we run out of room on the destination device,
we cannot recover the space from glance. glance-api will have open
unlinked files, so a TONNE of space is unavailable until we restart
glance-api.

Nova will try to reschedule the instance 3 times, so should see this nova-conductor.log :
u'RescheduledException: Build of instance 98ca2c0d-44b2-48a6-b1af-55f4b2db73c1 was re-scheduled: [Errno 28] No space left on device\n']

The problem is this code in
nova.image.glance.GlanceImageService.download():

        if data is None:
            return image_chunks
        else:
            try:
                for chunk in image_chunks:
                    data.write(chunk)
            finally:
                if close_file:
                    data.close()

image_chunks is an iterator.  If we take an exception (like we can't
write the file because the filesystem is full) then we will stop
iterating over the chunks.  If we don't iterate over all the chunks then
glance will keep the file open.

** Affects: nova
     Importance: Undecided
     Assignee: Chris Friesen (cbf123)
         Status: New


** Tags: compute

** Changed in: nova
     Assignee: (unassigned) => Chris Friesen (cbf123)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1461678

Title:
  nova error handling causes glance to keep unlinked files open, wasting
  space

Status in OpenStack Compute (Nova):
  New

Bug description:
  When creating larger glance images (like a 10GB CentOS7 image), if we
  run into situation where we run out of room on the destination device,
  we cannot recover the space from glance. glance-api will have open
  unlinked files, so a TONNE of space is unavailable until we restart
  glance-api.

  Nova will try to reschedule the instance 3 times, so should see this nova-conductor.log :
  u'RescheduledException: Build of instance 98ca2c0d-44b2-48a6-b1af-55f4b2db73c1 was re-scheduled: [Errno 28] No space left on device\n']

  The problem is this code in
  nova.image.glance.GlanceImageService.download():

          if data is None:
              return image_chunks
          else:
              try:
                  for chunk in image_chunks:
                      data.write(chunk)
              finally:
                  if close_file:
                      data.close()

  image_chunks is an iterator.  If we take an exception (like we can't
  write the file because the filesystem is full) then we will stop
  iterating over the chunks.  If we don't iterate over all the chunks
  then glance will keep the file open.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1461678/+subscriptions