openstack team mailing list archive

["Daniel P. Berrange" <berrange@xxxxxxxxxx>]

In the Libvirt driver there is special-case code for LXC to deal with
the volume-attach functionality, since there is no "block device attach"
functionality in libvirt for LXC. The code in question was added in

  commit e40b659d320b3c6894862b87adf1011e31cbf8fc
  Author: Chuck Short <chuck.short@xxxxxxxxxxxxx>
  Date:   Tue Jan 31 20:53:24 2012 -0500

    Add support for LXC volumes.
    
    This introduces volume support for LXC containers in Nova.
    The way that this works is that when a device is attached to an
    LXC container is that, the xml is parsed to find out which device to
    connect to the LXC container, binds the device to the LXC container,
    and allow the device through cgroups.
    
    This bug fixes LP: #924601.
    
    Change-Id: I00b41426ae8354b3cd4212655ecb48319a63aa9b
    Signed-off-by: Chuck Short <chuck.short@xxxxxxxxxxxxx>

First a little background....

The way LXC works with Nova, is that the image file assigned to the instance
eg 

  /var/lib/nova/instances/instance-0000000e/disk

is exported via qemu-nbd, and then mounted on the host at

  /var/lib/nova/instances/instance-0000000e/rootfs


When libvirt starts the container it uses that directory as the root
filesystem. libvirt will *also* mount a private /dev, /dev/pts, /proc
and /sys for the container. This is all fine

Now, when using 'nova volume-attach':

  # nova volume-attach 05eb16df-03b8-451b-85c1-b838a8757736 a5ad1d37-aed0-4bf6-8c6e-c28543cd38ac /dev/sdf

nova will import an iSCSI LUN from the nova volume service, on the compute
node. The kernel will assign it the next free SCSI drive letter, in my
case '/dev/sdc'.

The libvirt nova driver will then do a mknod, using the volume name
passed to 'nova volume-attach'.
eg it will do

  mknod  /var/lib/nova/instances/instance-0000000e/rootfs/dev/sdf

this is where it has all gone horribly wrong...

  * The iSCSI LUN is completely randomly allocated, and unrelated to the
    block device name the user will give to 'nova volume-attach'. So there
    is no association between the /dev/sdf in the container and the
    /dev/sdc in the host, and you can't expect the caller of 'volume-attach'
    to be able to predict what the next assigned LUN will be on the host.

  * The  /var/lib/nova/instances/instance-0000000e/rootfs/dev/ directory
    where nova did the mknod is a completely different filesystem to
    the one seen by the container. The /dev in the container is a tmpfs
    that is never visible to the host, so a mknod in the host won't
    appear to the container.

AFAIK, there is no way to resolve either of these problems given the
current level kernel support for LXC, which is why libvirt has never
implemented block volume attach itself.

Thus I'm wondering how this LXC volume-attach code in Nova has ever
worked, or was tested ? My testing of Nova shows no sign of it working
today. Unless someone can demonstrate a flaw in my logic, I'm inclined
to simply revert this whole commit from Nova.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|