nova team mailing list archive

[Gustavo Niemeyer <gustavo.niemeyer@xxxxxxxxxxxxx>]

Hi Eric,

> My concern here is that while Redis (or other database solutions)
> can scale quite easily for our data set size, there are still response
> time and HA concerns. You can use local caching to help response time
> and run Redis slaves with promotion for HA, but this still seems a
> bit clunky. One of our design tenets is shared-nothing, but right
> now the data store is shared everywhere.

Have you had a look at ZooKeeper[1]? It's a not a storage solution,
and definitely not a SQL database replacement, if that's what you're
looking for.  But what you seem to need is a coordination service,
right?

Zookeeper is a generic coordination service based on the concept of an
observable filesystem abstraction with strong guarantees of ordering,
atomicity, etc. It's a really good foundation piece to build
distributed systems upon.  It will solve your HA concern very well,
will likely solve many of the MQ uses, and can be twisted to hold many
other roles, so probably worth a look.

It doesn't natively help with the shared-nothing, though (quite the
opposite of that).  It would be possible to architect around it, by
sharding based on accounts or similar, if the advantages are worth it.

[1] http://hadoop.apache.org/zookeeper/

-- 
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/blog
http://niemeyer.net/twitter