launchpad-dev team mailing list archive

[Julian Edwards <julian.edwards@xxxxxxxxxxxxx>]

On Tuesday 21 September 2010 13:51:23 Aaron Bentley wrote:
> On 09/21/2010 04:25 AM, Julian Edwards wrote:
> > On Friday 17 September 2010 14:51:33 Aaron Bentley wrote:
> >>> Cron jobs have served us well so far, but isn't it about time we
> >>> stopped using them and get RabbitMQ sorted out instead?
> >> 
> >> It's not clear what the best approach would be.  It sounds like you're
> >> suggesting an approach that would require us to write and maintain our
> >> own daemon, but there are solutions out there that provide their own
> >> daemons.
> > 
> > I'm not suggesting anything (yet).
> 
> Okay. From what you said, it sounded like you wanted to use RabbitMQ,
> and not something else, such as Gearman's built-in messaging, for
> dispatching tasks.

Yeah, RabbitMQ was the first thing that popped into my head, possibly because 
rabbitmq-server is already running as a Launchpad dependency on my laptop 
(WHY?) and blocks its shutdown.

> >> That's why I've been gathering requirements at
> >> https://dev.launchpad.net/Foundations/NewTaskSystem/Requirements to
> >> ensure we don't pursue a dead end.
> > 
> > I think this is great; however it doesn't diminish the separate need for
> > a way of passing messages between components.
> 
> Selecting a way of passing messages is a critical part of satisfying the
> New Task System requirements. Celery, Gearman and raw RabbitMQ are
> separate messaging approaches.  So out of this process, you will get a
> messaging solution, but I'm hesitant to use RabbitMQ for Jobs before we
> have selected our solution, because it may turn out to be waste.
> 
> I have been assuming that dispatching tasks (e.g. Jobs, Builds) was the
> primary use case for RabbitMQ. You say we have a separate need for
> passing messages between components.  Can you give me examples of ways
> you'd like to use RabbitMQ that aren't related to tasks?

Mostly hand-offs between different parts of the upload and publishing pipeline 
in Soyuz.  I guess you could equate them to tasks if you want to, but it's 
simpler for me to think of it as a hand-off.

J