openstack team mailing list archive

[Pádraig Brady <P@xxxxxxxxxxxxxx>]

On 03/16/2012 11:57 PM, Justin Shepherd wrote:
> 
> 
> On Mar 16, 2012, at 12:26, "Pádraig Brady" <P@xxxxxxxxxxxxxx> wrote:
> 
>> On 03/16/2012 04:11 PM, Jay Pipes wrote:
>>> Hi Stackers,
>>>
>>> So, in diagnosing a few things on TryStack yesterday, I ran into an interesting problem with snapshotting that I'm hoping to get some advice on.
>>>
>>> == The Problem ==
>>>
>>
>>> QEMU was unhelpfully returning a vague error message of "error while writing".
>>
>> That could be improved.
>> As an aside, since qemu-img is mainly dealing with large files,
>> it would be a prime candidate to call fallocate() from
>> to get good layout for the files and immediate feedback
>> if there isn't enough space.
>>
>> On a related note, I've a patch pending for after RC1
>> that should auto clean any of these partially written files:
>> https://review.openstack.org/#change,5442
>>
>>> As it turns out, the base operating system we install on our compute nodes in TryStack has a (very) small root partition
>>
>>> == Possible Solutions ==
>>>
>>> So, there are a number of solutions that we can work on here, and I'm wondering what the preference would be. Here are the solutions I have come up with, along with a no-brainer improvement to Nova that would help in diagnosing this problem:
>>>
>>> The no-brainer: Detect before attempting a snapshot that there is enough space on a device to perform the operation, and if not, throw a useful error message up the stack
>>
>> The space can change while writing, so you could still get the same error above.
>>
>>>
>>> Solutions to the disk space problem:
>>>
>>> (1) Silly Jay, change the damn size of the root partition in your PXE base OS install!
>>>
>>> Now, I'm no expert in creating customized base disk images, but from looking at the build_pxe_env.sh script in devstack [1], it seems pretty trivial to change the ramdisk_size parameter in the startup options to something larger than 2109600. We could do this and reimage the compute nodes one by one.
>>>
>>> (2) Make the location in which the snapshot is made configurable.
>>>
>>> Right now, as mentioned above, tempfile.mkdtemp() is used, which creates a directory in the user's TMPDIR (typically /tmp, which is usually on the root partition).
>>>
>>> We could add an option (--libvirt-snapshot-dir?) that would allow nova-compute to override where that snapshot is built.
>>>
>>> (3) Change the user (running nova-compute) TMPDIR setting to something different than /tmp on the root partition).
>>
>> I'd lean towards (3).
>> That's something that depends on the environment (as you've nicely demonstrated),
>> and also for security reasons the admin should be able to set TMPDIR.
>> That's the standard way to do it, and it works already (hopefully).
> 
> Actually I would argue that the best way to accomplish this would be option #2. That way an admin/operator has control over the location. Not manipulating this by messing around with a users environment variable.

Well one can set the TMPDIR in the init script for the service.
That's a fairly standard mechanism.

(2) is good though if you would ever want to separate
--libvirt-snapshot-dir from, $TMPDIR

Now I can definitely see the need for changing TMPDIR from /tmp
for Jay's reasons and /tmp being tmpfs by default on debian for example:
http://lists.debian.org/debian-devel/2011/11/msg00281.html
I'm not sure if you'd need to separate them?
Though I'm always biased towards avoiding new config variables.
I suppose one could argue you might want /tmp for small fast accesses,
and something large and separate for manipulating large files.

Now that I look at the existing nova uses of tmp dirs
to store/stage large images, I see existing config vars:

FLAGS.xenapi_sr_base_path  # xens default Storage Repo
FLAGS.image_decryption_dir # nova/image/s3.py

So if you were following that you would implement (2) with:

FLAGS.libvirt_snapshot_dir

There might be opportunity to merge all three to:

FLAGS.nova_image_staging_dir

cheers,
Pádraig.