openstack team mailing list archive

[Andrew Holway <a.holway@xxxxxxxxxxxx>]

Hey,

I had NFS over RDMA working but hit this bug with o_direct. I cannot remember the exact circumstances of this cropping up.

https://bugzilla.linux-nfs.org/show_bug.cgi?id=228

You can work around this with KVM by explicitly specifying the block size in the XML. I do not know how you would implement this in openstack. With difficulty I imagine :)

<qemu:commandline><qemu:arg value='-set'/><qemu:arg value='-set'/><qemu:arg value='-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,logical_block_size=4096'/></qemu:commandline>

In general RDMA is best avoided for production setups. NFSoverRDMA support has been dumped from Mellanox OFED and Redhat support is really choppy. I saw weird kernel panics here and there and other general unhappiness.

ipoib is generally fast enough as long as you have it in connected mode and set the frame size appropriately try setting it a wee bit bigger than the block size of your filesystem.

ta for now

Andrew Holway



On Jan 10, 2013, at 10:14 PM, Mark Lehrer wrote:

> 
> Has anyone here been able to make Openstack + KVM work with Infiniband & NFS-RDMA?
> 
> I have spent a couple of days here trying to make it work, but with no luck.  At first I thought the problem was NFS3 and lockd, but I tried NFSv4 and I have the same problem.  I also disabled AppArmor just as a test but that didn't help.
> 
> Things work fine with Infinband if I don't use RDMA, and I have many non-libvirt QEMU-KVM vm's working fine using RDMA.
> 
> When this problem occurs, the qemu process is CPU locked, strace shows calls to FUTEX, select, and lots of EAGAIN messages.  If nobody else here is using this setup, I'll keep digging to find out exactly which option is causing this problem.
> 
> Thanks,
> Mark
> 
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp