launchpad-dev team mailing list archive

[Michael Hudson <michael.hudson@xxxxxxxxxxxxx>]

Bjorn Tillenius wrote:
> On Thu, Oct 08, 2009 at 04:02:02PM -0400, Aaron Bentley wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> Gary Poster wrote:
>>> Hi Tim and Michael.  I don't want Foundations to work on a project
>>> unless a team is going to be ready to consume it very soon.  We did have
>>> RabbitMQ on our plate, but neither Soyuz nor UI work appear to be able
>>> to use it this six month period.  Will you all be able to use that in
>>> these six months?
>> We want to generate merge proposals, diffs, branch upgrades, and emails
>> on demand.  For this, we need to be able to run Jobs on demand.  In
>> order to run Jobs on demand, we need RabbitMQ.  We have been eagerly
>> anticipating RabbitMQ since the Epic.
> 
> My question would be, why do you need to run Jobs on demand; what's the
> end-user visible change here?

If you've ever propose a branch for merging when the clock is a round 5
minutes past the hour and then had to wait for nearly 5 minutes for the
diff to appear -- that's what we want to fix.

(And other related things, of course).

> When we know that, is there maybe some
> other change we can do, instead of bringing in yet another moving part
> to Launchpad?
> 
> For example, how often do you poll today? Can we increase the frequency,
> so that it is like on-demand.

Of course, we can always poll more often.  There comes a limit though
where the mere act of polling frequently enough to get the
responsiveness we desire places an unacceptable load on the system.

As Aaron says, the fact that it takes program using Launchpad a good
number of seconds start up means that the simple approach we use for
polling doesn't scale to polling much more than once a minute (and also
the fact that cron doesn't naturally let you run more than once a minute!).

This problem is solvable too, with a long running polling process and a
process pool of workers perhaps.  The branch puller is sort of moving in
this direction.

> Also, I don't know that much how RabbitMQ works. If you have that, how
> will RabbitMQ start the right Job?

The most obvious thing would be to encode the job type and the job
database id in the message, I guess.

Cheers,
mwh