openstack team mailing list archive

[Chuck Short <chuck.short@xxxxxxxxxxxxx>]

On Thu, 5 Jul 2012 15:00:26 +0100
"Daniel P. Berrange" <berrange@xxxxxxxxxx> wrote:

> In the Libvirt driver there is special-case code for LXC to deal with
> the volume-attach functionality, since there is no "block device
> attach" functionality in libvirt for LXC. The code in question was
> added in
> 
>   commit e40b659d320b3c6894862b87adf1011e31cbf8fc
>   Author: Chuck Short <chuck.short@xxxxxxxxxxxxx>
>   Date:   Tue Jan 31 20:53:24 2012 -0500
> 
>     Add support for LXC volumes.
>     
>     This introduces volume support for LXC containers in Nova.
>     The way that this works is that when a device is attached to an
>     LXC container is that, the xml is parsed to find out which device
> to connect to the LXC container, binds the device to the LXC
> container, and allow the device through cgroups.
>     
>     This bug fixes LP: #924601.
>     
>     Change-Id: I00b41426ae8354b3cd4212655ecb48319a63aa9b
>     Signed-off-by: Chuck Short <chuck.short@xxxxxxxxxxxxx>
> 
> First a little background....
> 
> The way LXC works with Nova, is that the image file assigned to the
> instance eg 
> 
>   /var/lib/nova/instances/instance-0000000e/disk
> 
> is exported via qemu-nbd, and then mounted on the host at
> 
>   /var/lib/nova/instances/instance-0000000e/rootfs
> 
> 
> When libvirt starts the container it uses that directory as the root
> filesystem. libvirt will *also* mount a private /dev, /dev/pts, /proc
> and /sys for the container. This is all fine
> 
> Now, when using 'nova volume-attach':
> 
>   # nova volume-attach 05eb16df-03b8-451b-85c1-b838a8757736
> a5ad1d37-aed0-4bf6-8c6e-c28543cd38ac /dev/sdf
> 
> nova will import an iSCSI LUN from the nova volume service, on the
> compute node. The kernel will assign it the next free SCSI drive
> letter, in my case '/dev/sdc'.
> 
> The libvirt nova driver will then do a mknod, using the volume name
> passed to 'nova volume-attach'.
> eg it will do
> 
>   mknod  /var/lib/nova/instances/instance-0000000e/rootfs/dev/sdf
> 
> this is where it has all gone horribly wrong...
> 
>   * The iSCSI LUN is completely randomly allocated, and unrelated to
> the block device name the user will give to 'nova volume-attach'. So
> there is no association between the /dev/sdf in the container and the
>     /dev/sdc in the host, and you can't expect the caller of
> 'volume-attach' to be able to predict what the next assigned LUN will
> be on the host.
> 
>   * The  /var/lib/nova/instances/instance-0000000e/rootfs/dev/
> directory where nova did the mknod is a completely different
> filesystem to the one seen by the container. The /dev in the
> container is a tmpfs that is never visible to the host, so a mknod in
> the host won't appear to the container.
> 
> AFAIK, there is no way to resolve either of these problems given the
> current level kernel support for LXC, which is why libvirt has never
> implemented block volume attach itself.
> 
> Thus I'm wondering how this LXC volume-attach code in Nova has ever
> worked, or was tested ? My testing of Nova shows no sign of it working
> today. Unless someone can demonstrate a flaw in my logic, I'm inclined
> to simply revert this whole commit from Nova.
> 
> Regards,
> Daniel

Hi,

It *was* working at one point. Its on my todo list to make sure that it
still works properly. Otherwise Ill remove it myself.

Regards
chuck