launchpad-dev team mailing list archive

[Aaron Bentley <aaron@xxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 11-01-05 10:23 AM, Julian Edwards wrote:
> On Wednesday 05 January 2011 15:01:36 Aaron Bentley wrote:
>> You are focusing on a particular kind of solution to the "interruptible
>> processing of continuous data", but there are other kinds of solutions.
>>  Shouldn't we look at them, too?
>>
>>> Micro jobs are a nice idea, but are orthogonal to what I want to do here.
>>> They might even end up using this design.
>>
>> They provide a way of doing interruptible processing of continuous data.
>>  They only seem orthagonal because you're focusing on solutions that
>> store restart points.
> 
> I am open to your suggestions of a way of recovering from a data point (after 
> a crash or interval between cron jobs) without storing where you left off.  I 
> cannot see how the loop tuner nor micro jobs do that.

In my earlier example, each data record that needs to be processed would
have an associated micro-job.  The BuildStatus-like enum would indicate
whether the job has completed.

To recover, you just query for jobs that haven't been completed.  (If
you need to pick up exactly where you left off, you can sort the jobs or
records.)

> I am working on the basis that the client knows best and I just want to 
> provide a level of abstraction at a good layer.  The restart point would need 
> to be updated as often as the client felt necessary, which might be every time 
> it "started working on a new record."

Cool.  I initially feared that this wouldn't support multiprocessing.
It seems that it does, though of course more sync is required.

> Anyway this might all be moot, I'm going to look at the state of Rabbit and 
> work out what I can do with it.  I am hoping that it has some sort of unicast 
> style queue, with a recovery method for missed messages.

Using a queue would be another way to implement micro-jobs :-)

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0kj+0ACgkQ0F+nu1YWqI1DHgCcC3ANPdxedVvKX2I+0ZCU2vi7
kykAn3LD5kfg0aRNNv/thJlFnrQK3lWT
=3yMs
-----END PGP SIGNATURE-----