openstack team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

On Tue, Nov 15, 2011 at 3:03 PM, Ewan Mellor <Ewan.Mellor@xxxxxxxxxxxxx> wrote:
> Hi,
>
> What are the expected semantics for the Glance API when uploading an image
> who’s size you do not know?  The docs at
> http://glance.openstack.org/glanceapi.html say “x-image-meta-size: This
> header is optional. … When not present, Glance will calculate the image’s
> size based on the size of the request body.”  I read this as “If
> x-image-meta-size is missing, then Glance will accept the upload, and then
> update the metadata to match whatever what was sent”.  However (with
> Glance-over-Swift at least) that’s not what’s happening.  What’s actually
> happening is “If x-image-meta-size is missing, then the Content-Length
> header is read, and if both are missing then the image size is set to 0.”

What version of Glance are you using?

Also, how are you uploading the image to Glance?

> I’m not sure whether I should clarify the docs to say that x-image-meta-size
> is required if you don’t set Content-Length, or whether the Swift backend
> needs to be updated to record / recover the uploaded image size.

The X-Image-Meta-Size header is only used in the "image registration"
process -- i.e. when you are reserving an image identifier and want to
set some metadata about the image you plan to upload.

When the actual image is uploaded (either in the POST /images or PUT
/images/<IMAGE_ID>), the block of code that calculates what expected
size to pass to the backend store is this:

            if req.content_length:
                image_size = int(req.content_length)
            elif 'x-image-meta-size' in req.headers:
                image_size = int(req.headers['x-image-meta-size'])
            else:
                logger.debug(_("Got request with no content-length and no "
                               "x-image-meta-size header"))
                image_size = 0
            location, size, checksum = store.add(image_meta['id'],
                                                 req.body_file,
                                                 image_size)

The else: block is ONLY met when you are not using the Python glance
client, the glance CLI tool, and are not setting either the
Content-Length or X-Image-Meta-Size header. If you use the Python
glance client or CLI tool, the image you are feeding to the client
automatically does a seek/tell to determine the size of the image you
are uploading.

The only reason that the image_size is passed to the backend store is
because of Swift's inability to automatically handle files larger than
5GB.

The image_size passed to the swift backend store's add() method is
used to determine if the image file is larger than a configurable size
(defaults to 5GB). If it is larger, the backend store writes the image
file in chunks and manually uploads those segments to Swift, writing
the manifest file on successful upload to Swift. Unfortunately, if an
image_size of 0 is given, we have no choice but to attempt to write
the image file to Swift directly and hope the image file is not
greater than Swift's maximum (before manifest-ing) object size. The
alternative would be to have to write the image file to disk on the
Glance API node for every single upload to Swift where image_size was
0, essentially doubling the writes for any image files not greater
than 5GB.

My guess is that you are either using an older version of the Glance
client library (that doesn't seek/tell to set the Content-Length
header) or you are using a non-standard way of uploading (Java
perhaps? ;)

Cheers,
-jay