openstack team mailing list archive

[Brian Schott <brian.schott@xxxxxxxxxxxxxxxxxx>]

You did see that Amazon hit #42 on the Top 500 supercomputing list? It is somewhat of a stunt, but the point is that access to a supercomputer is a credit-card swipe away and rentable by the hour.  There was a lot of buzz at SC11 a few weeks ago.

There are several HPC groups in the OpenStack community:
- The DOE Magellan OpenStack system is intended for mid-range HPC workloads.
- My former group at USC/ISI joined OpenStack during Bexar to deploy on large shared memory HPC systems (SGI UltraViolet 1TB of main memory), heterogeneous cluster computing (GPU accelerators, many-core processor architectures like Tilera), and tightly coupled cluster applications over InfiniBand and/or 10GbE).  The USC-ISI team is still carrying on that work.  
- At Nimbis, I'm focused on technical computing workloads for companies that lack access to HPC.  We work with traditional HPC centers like NCSA, OSC, and R-Systems, but many of the configuration management and tenant isolation issues we encounter dealing with small users in traditional PBS/Moab batch systems would be easier if these centers ran OpenStack.

The challenges for virtualization on HPC are mostly focused on the I/O subsystem because there is a lot of highly tuned hardware for high-end networking, disk array subsystems, hardware accelerators and they don't know about virtual machines generally.  If you have an MPI offload engine running in your network card, it expects to pair with a single kernel, not a host and a guest.  Exposing these devices through Xen or KVM can be difficult even if you don't try to share the devices across VMs.  LXC is a reasonable approach but you lose some of the flexibility and isolation of true VMs.

The things that OpenStack can focus on are things that we've created blueprints for:
- alternative VMs like LXC from the scheduler
- consideration for bare-metal provisioning where you move vlan management into the switch
- cluster-level schedulers that take account of network topology requirements, bandwidth, latency, hops
- scheduler support for non-x86 and x86+extra hardware

Having said that, the OpenStack architecture is ideal for folks that want to bridge the gap between cloud and HPC.  The community is vibrant and moving fast and the architecture is flexible enough to allow many different use cases by design.  It's a meritocracy where code wins, which is why I like it.   I spent a lot of time at SC11 talking to HPC folks about OpenStack.  

Brian

-------------------------------------------------
Brian Schott, CTO
Nimbis Services, Inc.
brian.schott@xxxxxxxxxxxxxxxxxx
ph: 443-274-6064  fx: 443-274-6060







On Dec 2, 2011, at 9:18 AM, Oleg Gelbukh wrote:

> Hello,
> 
> Here at Mirantis we are working on deployment of Openstack that intended to manage HPC cluster eventually. There are few features that we are going to incorporate, and we are still researching. The general idea is to use LXC as a lightweight virtualization engine, and make use of faster I/O system than that based on disk image file.
> 
> --
> Oleg Gelbukh,
> Sr. IT Engineer
> Mirantis Inc.
> 
> On Fri, Dec 2, 2011 at 4:17 PM, Sandy Walsh <sandy.walsh@xxxxxxxxxxxxx> wrote:
> I've recently had inquiries about High Performance Computing (HPC) on Openstack. As opposed to the Service Provider (SP) model, HPC is interested in fast provisioning, potentially short lifetime instances with precision metrics and scheduling. Real-time vs. Eventually.
> 
> Anyone planning on using Openstack in that way?
> 
> If so, I'll direct those inquires to this thread.
> 
> Thanks in advance,
> Sandy
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp