openstack team mailing list archive

[Pádraig Brady <P@xxxxxxxxxxxxxx>]

On 09/08/2012 11:29 AM, Rich Wohlstadter wrote:
> Hi everyone,
>
> I'm running into an issue where I'm using nfs on my multi-node setup for
> /var/lib/nova/instances so that I can use live migration. Things work
> great unless I try to spawn multiple vm instances simultaneously and the
> image does not yet exist in the _base directory. I will get multiple
> spawning errors on a few of the instances that are trying to spawn,
> usually with the following error: OSError: [Errno 2] No such file or
> directory:
> '/var/lib/nova/instances/_base/1cab2bf14012771017e66501a98bf594877c9d2d.part'.
> It looks like multiple nodes are trying to do things on the same file and
> 1 node is deleting the file while the others are trying to access it thus
> producing this error. Once the image and the resized image based on flavor
> is out there and present in the _base directory then I no longer get
> errors when spawning multiple instances at the same time. So basically it
> seems to be an issue only the first time an image has to be copied over
> from glance and resized. Anyone seen this or is this a known issue when
> running multi-node instance directory over nfs and/or glusterfs?

There are two ways to support this.

https://review.openstack.org/#/q/Icff10ed75ba83f7256731614dc9e01e578b347a4,n,z
This older way was to have a separate base directory per compute node,
which you enable by doing this in nova.conf:
  base_dir_name=_base_$my_ip
It was suggested in that review (on master) that a shared lock dir could work
also, along with some potential issues with doing that.

https://review.openstack.org/#/q/Ic2ac87840904fa199c17774dae9556ad6c7a3eaf,n,z
only just implemented on master, implements the shared lock dir for a shared
instance directory implicitly. You can still use the config var in the
first method with this in place, (which might be required if you were running
into serialization issues between compute nodes).

cheers,
Pádraig.