launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Sat, Jun 4, 2011 at 2:16 AM, Thomas Hervé <thomas.herve@xxxxxxxxxxxxx> wrote:
> Le 03/06/2011 04:33, Robert Collins a écrit :
>
> Hi Robert, thanks for sharing the ideas. I have one slightly unrelated
> discussion point:
>
>> Rabbit has an awkward high availability story; specifically its not
>> trivial to get the reliability we have out of HTTP services, This is
>> partly because rabbit clusters don't distribute the queues and because
>> its a more stateful and complex system than HTTP.
>
> I've been think about that, and I wonder how much we could cheat for
> some specific workloads: if you have "short-lived" queues, I was
> thinking that the application (or a smart proxy) could publish messages
> to all the different RabbitMQ instances available.
>
> Consumers would then consume from *one* of those RabbitMQ instance, and
> the others would expire after some time. You'd probably have to manage
> duplicate messages on the consumer side, but that's generally the case
> anyway.
>
> It may be a lot of work to workaround the shortcomings of RabbitMQ you
> mention, though.

Indeed. Contrast this with HTTP: we configure out HTTP service point
to be HA, and a node failure takes out in-progress messages only with
no specific code needed in each client. With rabbit all our clients
and most/all of our consumers would need to know about the HA
arrangements. That seems undesirable.

Now, its probably possible to write a wrapping layer that sits on
amqp, takes one message, multiplies it, and sends, and conversely one
that sits attached to two rabbits listening and gets advisory locks
etc, or that ensures all consumers are pulling from just one rabbit.
Yeouch.

@Aaron burrow looks interesting. They don't /seem/ to be designing HA
in (which isn't necessarily bad, but isn't necessarily good either).

Some other queueing systems are around that are pretty interesting
like 0mq and kestrel.

I'm not sure what we should do queue wise; I'm inclined to stick with
Rabbit until its either too much bother, or something massively better
(e.g. massively simpler with sameish facilities, or similar complexity
and more facilities (like HA)) comes along.

-Rob