openstack team mailing list archive

["Daniel P. Berrange" <berrange@xxxxxxxxxx>]

On Thu, Jul 05, 2012 at 03:00:26PM +0100, Daniel P. Berrange wrote:
> Now, when using 'nova volume-attach':
> 
>   # nova volume-attach 05eb16df-03b8-451b-85c1-b838a8757736 a5ad1d37-aed0-4bf6-8c6e-c28543cd38ac /dev/sdf
> 
> nova will import an iSCSI LUN from the nova volume service, on the compute
> node. The kernel will assign it the next free SCSI drive letter, in my
> case '/dev/sdc'.
> 
> The libvirt nova driver will then do a mknod, using the volume name
> passed to 'nova volume-attach'.
> eg it will do
> 
>   mknod  /var/lib/nova/instances/instance-0000000e/rootfs/dev/sdf

Opps, I'm slightly wrong here. What it actually does is

  mount --bind /dev/sdc /var/lib/nova/instances/instance-0000000e/rootfs/dev/sdf

so you get a 'sdf' device, but with the major/minor number of the 'sdc'
device. I can't say I particularly like this approach. Ultimately I
think we need the kernel support to make this work correctly. In any
case, even using mount --bind, doesn't deal with the fact that the guest'
/dev is not visible from the host

> this is where it has all gone horribly wrong...
> 
>   * The iSCSI LUN is completely randomly allocated, and unrelated to the
>     block device name the user will give to 'nova volume-attach'. So there
>     is no association between the /dev/sdf in the container and the
>     /dev/sdc in the host, and you can't expect the caller of 'volume-attach'
>     to be able to predict what the next assigned LUN will be on the host.
> 
>   * The  /var/lib/nova/instances/instance-0000000e/rootfs/dev/ directory
>     where nova did the mknod is a completely different filesystem to
>     the one seen by the container. The /dev in the container is a tmpfs
>     that is never visible to the host, so a mknod in the host won't
>     appear to the container.
> 
> AFAIK, there is no way to resolve either of these problems given the
> current level kernel support for LXC, which is why libvirt has never
> implemented block volume attach itself.
> 
> Thus I'm wondering how this LXC volume-attach code in Nova has ever
> worked, or was tested ? My testing of Nova shows no sign of it working
> today. Unless someone can demonstrate a flaw in my logic, I'm inclined
> to simply revert this whole commit from Nova.
> 

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|