openstack team mailing list archive

[Jay Pipes <jaypipes@xxxxxxxxx>]

On Tue, Nov 15, 2011 at 5:26 PM, Ewan Mellor <Ewan.Mellor@xxxxxxxxxxxxx> wrote:
>> From: Ewan Mellor
>> > From: Jay Pipes [mailto:jaypipes@xxxxxxxxx]
>> > > From: Ewan Mellor
>> > > ~ # cat test_glance.py
>> > > import sys
>> > > import glance.client
>> > >
>> > > client = glance.client.Client('localhost', 9292,
>> > auth_tok="999888777666")
>> > > print client.add_image({}, sys.stdin)
>> > > ~ # echo a | python26 ./test_glance.py
>> > > {u'status': u'active', u'name': None, u'deleted': False,
>> > u'container_format': None, u'created_at': u'2011-11-15T21:44:21',
>> > u'disk_format': None, u'updated_at': u'2011-11-15T21:44:22', u'id':
>> 6,
>> > u'owner': u'Administrator', u'location':
>> > u'swift+http://root:password@localhost:5000/v1.0/glance/6',
>> u'min_ram':
>> > 0, u'checksum': u'60b725f10c9c85c70d97880dfe8191b3', u'min_disk': 0,
>> > u'is_public': False, u'deleted_at': None, u'properties': {}, u'size':
>> > 0}
>> > >
>> > > Note that size is returned as 0, not 1.
>> >
>> > Yes, indeed. That is because the client is not designed to be used
>> > that way [snip]
>>
>> But it could perfectly well be used this way, modulo the bug above, and
>> this is a very useful thing to be able to do (stdin here could be a
>> decompression or decryption pipeline, or a read from a remote socket,
>> or
>> whatever.
>>
>> With the bug in the Swift backend fixed, I think it will work just fine
>> to stream through a glance client in this way.
>
> Erm, well it will until we hit the 5GB Swift chunking limit, anyway.
>
> Do you know what this means (glance/store/swift.py +347)?
> Why can't we stream from body_file?
>
>            else:
>                # Write the image into Swift in chunks. We cannot
>                # stream chunks of the webob.Request.body_file, unfortunately,
>                # so we must write chunks of the body_file into a temporary
>                # disk buffer, and then pass this disk buffer to Swift.

We can (and we do in the case when image_size=0 when calling the Swift
backend store's add() method). Unfortunately, since you don't know the
file size, you begin streaming to Swift (essentially by passing
body_file on to the swift.client.Connection.put_object() method).
Unfortunately, if the file is >5GB, what will happen is that Swift
will vomit after attempting to upload 5G of data, resulting in a
situation where the Glance Swift storage driver would then have to try
to restart the upload using a segmented approach. But in order to do
that, the driver would have to seek(0) back on the body_file.

However, by default, webob.Request.body_file is NOT seekable. To make
it seekable, you need to either do this:

webob.Request.make_body_seekable()

or this:

webob.Request.is_body_seekable = True

Unfortunately, doing either of the above results in the entire request
body being buffered to disk on the server side in order for webob to
convert the (unseekable) StringIO it uses for webob.Request.body into
a (seekable) regular file object that it then assigns to its
webob.Request.body_file attribute.

So, let's say we tried to handle re-trying a segmented upload after we
first try to upload a 6GB image file *without* setting the image_size
to the actual size. What would happen is this:

Write 5GB of data to the swift socket
Get error, make body_file seekable in order to seek(0) back to the
request body's starting point, which results in copying the entire
body_file (6GB) into a temporary file on the server
seek(SEEK_END) on that body_file to get the real image size, and then
stream 6GB of data in chunks to Swift using its segmented upload
method.

Total data written to disk and network for 6GB image file: 5GB + 6GB +
6GB = 17GB

Or you can set the image_size properly by determining the file size
ahead of time. :)

You can read more about this issue here:

https://bugs.launchpad.net/glance/+bug/794718
https://bugs.launchpad.net/glance/+bug/818292
upstream: https://bitbucket.org/ianb/webob/issue/12/fix-for-issue-6-broke-chunked-transfer

Cheers,
-jay