launchpad-dev team mailing list archive

[Jamu Kakar <jkakar@xxxxxxxx>]

Hi,

On Fri, Jun 3, 2011 at 10:27 PM, Robert Collins
<robertc@xxxxxxxxxxxxxxxxx> wrote:
> On Sat, Jun 4, 2011 at 2:16 AM, Thomas Hervé <thomas.herve@xxxxxxxxxxxxx> wrote:
>> Le 03/06/2011 04:33, Robert Collins a écrit :
>> Hi Robert, thanks for sharing the ideas. I have one slightly unrelated
>> discussion point:
>>
>>> Rabbit has an awkward high availability story; specifically its not
>>> trivial to get the reliability we have out of HTTP services, This is
>>> partly because rabbit clusters don't distribute the queues and because
>>> its a more stateful and complex system than HTTP.
>>
>> I've been think about that, and I wonder how much we could cheat for
>> some specific workloads: if you have "short-lived" queues, I was
>> thinking that the application (or a smart proxy) could publish messages
>> to all the different RabbitMQ instances available.
>>
>> Consumers would then consume from *one* of those RabbitMQ instance, and
>> the others would expire after some time. You'd probably have to manage
>> duplicate messages on the consumer side, but that's generally the case
>> anyway.
>>
>> It may be a lot of work to workaround the shortcomings of RabbitMQ you
>> mention, though.
>
> Indeed. Contrast this with HTTP: we configure out HTTP service point
> to be HA, and a node failure takes out in-progress messages only with
> no specific code needed in each client. With rabbit all our clients
> and most/all of our consumers would need to know about the HA
> arrangements. That seems undesirable.
>
> Now, its probably possible to write a wrapping layer that sits on
> amqp, takes one message, multiplies it, and sends, and conversely one
> that sits attached to two rabbits listening and gets advisory locks
> etc, or that ensures all consumers are pulling from just one rabbit.
> Yeouch.
>
> @Aaron burrow looks interesting. They don't /seem/ to be designing HA
> in (which isn't necessarily bad, but isn't necessarily good either).
>
> Some other queueing systems are around that are pretty interesting
> like 0mq and kestrel.
>
> I'm not sure what we should do queue wise; I'm inclined to stick with
> Rabbit until its either too much bother, or something massively better
> (e.g. massively simpler with sameish facilities, or similar complexity
> and more facilities (like HA)) comes along.

I can't help but wonder if skipping RabbitMQ for the reasons above is
going too far.  High availability is important, especially for the
plumbing that connects everything together, but I wonder how much of
it you really need?  It seems the benefits of a queue based model are
many and that using HTTP or protobufs will result in an architecture
that, while it may be (more) highly available, will involve all kinds
of other deployment, design and performance hassles.

That said, I don't have enough experience with RabbitMQ to say,
"you're thinking about this wrong", or conversely, "yes, this is a
serious issue".  I am slightly concerned that overengineering could
lead to a suboptimal solution.  The impression I get from the outside,
and maybe I'm totally off the mark, is that Launchpad often chooses a
hard path to Do Things Right(tm), and then the end result is that
everything is hard.

I also wonder how many of these services need to talk to each other.
Maybe you could run many RabbitMQ instances and use them for
particular tasks?  For example, a bug-focused queue for bug-related
operations, a code hosting-focused queue for code-related operations,
etc.  If one of them falls over you end up with degraded service, as
opposed to losing everything.  I don't know how viable that is, since
I don't really understand what the topology of micro-services will
look like.

Also, is there something that will solve the HA issues you've brought
up in the pipeline for RabbitMQ?  Maybe it's something worth
contributing to and/or living without for some time while support for
these issues gets baked in?

How do other people use RabbitMQ and sleep at night?

Thanks,
J.