launchpad-dev team mailing list archive

[Aaron Bentley <aaron@xxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/17/2010 04:55 AM, Julian Edwards wrote:
> On Friday 17 September 2010 00:21:24 Tim Penhey wrote:
>> On Fri, 17 Sep 2010 11:05:22 Edwin Grubbs wrote:
>>> Since the job queue system just requires a run() method to do the
>>> actual work, I'm surprised that there isn't a cron script that runs
>>> any job type that doesn't opt to use its own cron script. As it is
>>> now, any new job table needs a new script, and that means a new
>>> database user also has to be added. Is there any reason we don't have
>>> a generic cron script already? A single cron script would also make it
>>> easier to control how many jobs are being run in parallel, and that
>>> would help prevent overloading the system, which would in turn help
>>> accurately estimating the time till a job is finished.
>>>
>>> -Edwin
>>
>> Over time the job runner has become more and more generic.
>>
>> Aaron and I have discussed the ability to have a generic job running cron
>> script now that almost everything could be configured through the command
>> line. We agreed that it is now possible and even desirable, but we only
>> have square tuits.
> 
> Cron jobs have served us well so far, but isn't it about time we stopped using 
> them and get RabbitMQ sorted out instead?

It's not clear what the best approach would be.  It sounds like you're
suggesting an approach that would require us to write and maintain our
own daemon, but there are solutions out there that provide their own
daemons.

That's why I've been gathering requirements at
https://dev.launchpad.net/Foundations/NewTaskSystem/Requirements to
ensure we don't pursue a dead end.

> It would help me greatly to reduce 
> startup latency (amongst other things) in loads of places now.

Absolutely.  As would gearman, or celery, etc.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkyTcmUACgkQ0F+nu1YWqI1VmQCfclEZeTy9AuNfXaCDszhEloWx
eeEAnA8tT4KbVesdhfN3rePRRLcPV3+I
=vnfy
-----END PGP SIGNATURE-----