openstack team mailing list archive

[Pádraig Brady <P@xxxxxxxxxxxxxx>]

On 03/16/2012 04:11 PM, Jay Pipes wrote:
> Hi Stackers,
> 
> So, in diagnosing a few things on TryStack yesterday, I ran into an interesting problem with snapshotting that I'm hoping to get some advice on.
> 
> == The Problem ==
> 

> QEMU was unhelpfully returning a vague error message of "error while writing".

That could be improved.
As an aside, since qemu-img is mainly dealing with large files,
it would be a prime candidate to call fallocate() from
to get good layout for the files and immediate feedback
if there isn't enough space.

On a related note, I've a patch pending for after RC1
that should auto clean any of these partially written files:
https://review.openstack.org/#change,5442

> As it turns out, the base operating system we install on our compute nodes in TryStack has a (very) small root partition

> == Possible Solutions ==
> 
> So, there are a number of solutions that we can work on here, and I'm wondering what the preference would be. Here are the solutions I have come up with, along with a no-brainer improvement to Nova that would help in diagnosing this problem:
> 
> The no-brainer: Detect before attempting a snapshot that there is enough space on a device to perform the operation, and if not, throw a useful error message up the stack

The space can change while writing, so you could still get the same error above.

> 
> Solutions to the disk space problem:
> 
> (1) Silly Jay, change the damn size of the root partition in your PXE base OS install!
> 
> Now, I'm no expert in creating customized base disk images, but from looking at the build_pxe_env.sh script in devstack [1], it seems pretty trivial to change the ramdisk_size parameter in the startup options to something larger than 2109600. We could do this and reimage the compute nodes one by one.
> 
> (2) Make the location in which the snapshot is made configurable.
> 
> Right now, as mentioned above, tempfile.mkdtemp() is used, which creates a directory in the user's TMPDIR (typically /tmp, which is usually on the root partition).
> 
> We could add an option (--libvirt-snapshot-dir?) that would allow nova-compute to override where that snapshot is built.
> 
> (3) Change the user (running nova-compute) TMPDIR setting to something different than /tmp on the root partition).

I'd lean towards (3).
That's something that depends on the environment (as you've nicely demonstrated),
and also for security reasons the admin should be able to set TMPDIR.
That's the standard way to do it, and it works already (hopefully).

cheers,
Pádraig.