launchpad-dev team mailing list archive

[Aaron Bentley <aaron@xxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/22/2010 09:40 AM, Elliot Murphy wrote:
>> On 09/21/2010 09:32 PM, Steve McInerney wrote:
>>> On Tue, 2010-09-21 at 08:51 -0400, Aaron Bentley wrote:
>>>> Okay. From what you said, it sounded like you wanted to use RabbitMQ,
>>>> and not something else, such as Gearman's built-in messaging, for
>>>> dispatching tasks.
>>>
>>> fwiw, rabbitmq is in use by both LS and U1, if you're unaware?
>>
>> Yes, I am aware, and that's certainly a factor we should consider when
>> choosing our approach.  But we shouldn't let it interfere with choosing
>> the best approach for us.
> 
> After reading this thread I still am not understanding why the
> hesitation around RabbitMQ (and I am pleased that you are pushing for
> the "best approach").

We have multiple systems that perform tasks in Launchpad, including the
build farm, the Job system and the code import system.  I think that we
can, and should, unify these systems.  In order to ensure that the
outcome is a fit replacement for all of them, I am gathering
requirements.
https://dev.launchpad.net/Foundations/NewTaskSystem/Requirements

My reading of Julian's message was that he wanted us to use RabbitMQ
directly, not running on top of Celery or anything else.  (He has since
said that wasn't what he meant, but that's what I was responding to.)
Since we are interested in the task dispatching domain, I expect that
solutions in that domain (Celery, Gearman) will require less effort and
maintenance than solutions in the message queueing domain (RabbitMQ) to
meet our needs.

> There are considerable costs such as integration
> with monitoring/backup/security systems to operating and maintaining
> each new piece of technology in our stack

In my experience, that is only sometimes true.  For example, we were
able to implement Ampoule into our Job stack with very little fuss.
(Ampoule is a Twisted library that supports remote execution of tasks
via Twisted's AMP messaging protocol.)

>, and one of the reasons that
> RabbitMQ was selected as a preferred technology at Canonical was
> because we were convinced that it was flexible enough to handle the
> varying needs from different product groups and we could get training
> and support on it (something that was not true of gearman at least at
> that time). If there is some capability missing from RabbitMQ core,
> there is a full time development team (recently acquired by
> VMWare/SpringSource) who are very interested in hearing feedback and
> feature requests from us.

These are factors that are certainly worth considering.

> RabbitMQ has been promoted to "main" in Ubuntu 10.04 Lucid, which
> means it is officially supported by Canonical for years to come, and I
> will be backporting RabbitMQ 2.0.1 to lucid-backports as soon as the
> 11.04 repositories open up next month - this version is noteworthy
> because it adds support for AMQP 0.9.1. RabbitMQ supports not only
> AMQP, but Stomp, JSON/RPC, SMTP, XMPP, PubSubHubBub, and more.
> http://www.rabbitmq.com/blog/2010/08/27/growing-up/

These are factors that are certainly worth considering.

> What does Launchpad need to do that is not handled by RabbitMQ?

I think it does not support many of the features listed here:
https://dev.launchpad.net/Foundations/NewTaskSystem/Requirements

Anything not listed here is something we must implement ourselves or
find another implementation of.  I think that Celery or Gearman are
already closer to what we need, so there is not as much for us to implement.

> Are
> there other libraries on top of RabbitMQ such as
> http://celeryq.org/docs/index.html that we should adopt in order to
> make it easier to use Rabbit?

At this point, I'm still gathering requirements, so I haven't looked at
all the potential solutions yet.  I think the three main contenders so
far are:
1. Celery on RabbitMQ
2. Gearman
3. The existing Launchpad job system, updated to accept RabbitMQ events,
run as a daemon, and provide a master/slave architecture for the build
farm and code imports.

> I'm keen to help make sure that the
> Launchpad needs are met elegantly and that an MQ system can be rapidly
> deployed for all the developers to build upon - I was tearing my hair
> out over the lack of a queueing system 2 years ago when I last worked
> on Launchpad so it's long overdue :)

Preach on, brother!  I originally implemented the Jobs system two years
ago with the understanding that we would soon be integrating RabbitMQ
into it (ala 3 above).  I've was quite disappointed that it took so long
for RabbitMQ to be made available to us.

But I don't think we should rush into 3.  I think that our needs are not
especially unique, and existing solutions may come pretty close to
offering what we need, even before we start building on top of them.  I
have trouble imagining that RabbitMQ, by itself, would be better for us
than running Celery on top of RabbitMQ.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkyaFWAACgkQ0F+nu1YWqI2AbwCdF9WNNI6MrvoWXonbkadw4gCs
XeUAnRnQ/W4PbUyOptRhqkuTeoqnFpP2
=WYdO
-----END PGP SIGNATURE-----