openstack team mailing list archive
Mailing list archive
Re: Queue Service, next steps
Eric & Team,
OpenStack's QueueService seems very interesting. As we have an existing
message queue implementation, we'd be happy to help you guys out. We're
about making messaging cloud-scale, so that everyone benefits.
However, it worries us that you're planning to implement a REST API for
messaging. Message queuing is fundamentally asynchronous; this is one of the
reasons StormMQ got started, as we found that approaches that use it (eg
SQS) suffer from some major weaknesses:-
- They're too slow;
- They can't handle sustained volumes
- Higher-level needs, eg fanout, selective pub-sub and transactions, are an
awkward, if not impossible, fit
There are a hoard of technical reasons why HTTP, superb as it is for
request-response architectures, makes a poor backbone for messaging (some of
the team behind StormMQ implemented one of the first banking-scale REST
For example, implementations that need to send or consume lots of data, and
are only interested in a subset whose filter criteria changes over time.
Syslogging, for example. Imagine a dynamic cloud, where servers come and go
- and centralised logging systems and alerts need no configuration, because
they use queuing. Under load (eg hack attempts on your server firewalls
generate 1000s of log messages) it mustn't fail, just go a bit slower.
StormMQ use AMQP internally for our own log management for that reason.
I read two things that surprised me on the OpenStack wiki. Firstly, that
AMQP is complicated and unsuitable for high-latency, unreliable links, and
secondly, that's why we have a REST API. I'd like to address that.
First up, AMQP isn't actually very complex at the level of an application
developer. Indeed, with a good library (like ours) it's trivially easy. The
apparent complexity comes becomes of unfamiliarity, both with concepts and
with use; no different to HTTP when it first came in (and we saw a plethora
of weird ways of using it and misunderstood criteria for headers, etc).
AMQP's highly suited to high-latency, unreliable links. That's why Smith
Electric vehicles use it to connect all their delivery trucks using dodgy 3G
links - and still gather 10,000s of items of data a second. The AMQP
protocol, particularly 1.0, make it's extremely clear how and when to
recover from failure. Indeed, AMQP's approach is failure happens - so deal
with it. HTTP on the other hand, has no such level of transactionality.
Second up, more importantly, StormMQ do not provide a REST API as an
alternative to AMQP. It's to provide features that are nothing to do with
message queuing - dynamically slicing up your cloud, for instance, or
managing environments to allow exact reproduceability or checking in to
source control your config. We'd be interested in providing a REST API if
there's the demand. AMQP does support multi-tenancy - we do it.
To assist, pragamatically, we'd like to donate as open source our upcoming C
and Java clients for AMQP 1.0, and help sponsor Python, Perl, PHP and Ruby
ones off the C code, so that there is as wide as possible opportunity for
people to use messaging.
I'd strongly encourage you to get involved in the AMQP working group so if
there's needs that are not met by AMQP, they can be addressed. The working
group is really keen to encourage an open, widely adopted standard for AMQP;
they'd like it to be the HTTP of messaging. Many of the features I see
proposed for OpenStack are features in AMQP - and AMQP has spent a lot of
time working out the kinks in edge cases and making sure they'd work with
the legacy - JMS, TIBCO and the like.
Message Queuing is easy to get into. Like chess, though, it can take a
lifetime to master.
We're by no means experts, but we're happy to help,
Gateshead int'l Business Centre, Mulgrave Terrace, Gateshead, NE8 1AN,
Telephone: +44 845 3712 567
78 Broomfield Road, Chelmsford, Essex, CM1 1SS, United Kingdom
StormMQ Limited is Registered in England and Wales under Company Number