openstack team mailing list archive

[Lorin Hochstein <lorin@xxxxxxxxxxxxxxxxxx>]

On Mar 16, 2012, at 7:57 PM, Justin Shepherd wrote:

> 
> 
> Sent from my iPad
> 
> On Mar 16, 2012, at 12:26, "Pádraig Brady" <P@xxxxxxxxxxxxxx> wrote:
> 
>> On 03/16/2012 04:11 PM, Jay Pipes wrote:
>>> Hi Stackers,
>>> 
>>> So, in diagnosing a few things on TryStack yesterday, I ran into an interesting problem with snapshotting that I'm hoping to get some advice on.
>>> 
>>> == The Problem ==
>>> 
>> 
>>> QEMU was unhelpfully returning a vague error message of "error while writing".
>> 

We ran into this problem in our Diablo deployment as well. It definitely needs a more informative error message.

>> 
>>> == Possible Solutions ==
>>> 
>>> So, there are a number of solutions that we can work on here, and I'm wondering what the preference would be. Here are the solutions I have come up with, along with a no-brainer improvement to Nova that would help in diagnosing this problem:
>>> 
>>> The no-brainer: Detect before attempting a snapshot that there is enough space on a device to perform the operation, and if not, throw a useful error message up the stack
>> 
>> The space can change while writing, so you could still get the same error above.
>> 
>>> 
>>> Solutions to the disk space problem:
>>> 
>>> (1) Silly Jay, change the damn size of the root partition in your PXE base OS install!
>>> 
>>> Now, I'm no expert in creating customized base disk images, but from looking at the build_pxe_env.sh script in devstack [1], it seems pretty trivial to change the ramdisk_size parameter in the startup options to something larger than 2109600. We could do this and reimage the compute nodes one by one.
>>> 
>>> (2) Make the location in which the snapshot is made configurable.
>>> 
>>> Right now, as mentioned above, tempfile.mkdtemp() is used, which creates a directory in the user's TMPDIR (typically /tmp, which is usually on the root partition).
>>> 
>>> We could add an option (--libvirt-snapshot-dir?) that would allow nova-compute to override where that snapshot is built.
>>> 
>>> (3) Change the user (running nova-compute) TMPDIR setting to something different than /tmp on the root partition).
>> 
>> I'd lean towards (3).
>> That's something that depends on the environment (as you've nicely demonstrated),
>> and also for security reasons the admin should be able to set TMPDIR.
>> That's the standard way to do it, and it works already (hopefully).
> 
> Actually I would argue that the best way to accomplish this would be option #2. That way an admin/operator has control over the location. Not manipulating this by messing around with a users environment variable.
> 



I agree with Pádraig that option #2 is a better way to go. I'd also recommend that it default to something under /var/lib/nova/instances, since that directory should generally be mounted a large partition by default.


Take care,

Lorin
--
Lorin Hochstein
Lead Architect - Cloud Services
Nimbis Services, Inc.
www.nimbisservices.com