launchpad-dev team mailing list archive

[Julian Edwards <julian.edwards@xxxxxxxxxxxxx>]

On Wednesday 05 January 2011 15:36:13 Aaron Bentley wrote:
> On 11-01-05 10:23 AM, Julian Edwards wrote:
> > On Wednesday 05 January 2011 15:01:36 Aaron Bentley wrote:
> >> You are focusing on a particular kind of solution to the "interruptible
> >> processing of continuous data", but there are other kinds of solutions.
> >> 
> >>  Shouldn't we look at them, too?
> >>  
> >>> Micro jobs are a nice idea, but are orthogonal to what I want to do
> >>> here. They might even end up using this design.
> >> 
> >> They provide a way of doing interruptible processing of continuous data.
> >> 
> >>  They only seem orthagonal because you're focusing on solutions that
> >> 
> >> store restart points.
> > 
> > I am open to your suggestions of a way of recovering from a data point
> > (after a crash or interval between cron jobs) without storing where you
> > left off.  I cannot see how the loop tuner nor micro jobs do that.
> 
> In my earlier example, each data record that needs to be processed would
> have an associated micro-job.  The BuildStatus-like enum would indicate
> whether the job has completed.
> 
> To recover, you just query for jobs that haven't been completed.  (If
> you need to pick up exactly where you left off, you can sort the jobs or
> records.)

On what data would you sort?  How do you work out what micro-jobs to create 
next by the thing that produces them?

> Using a queue would be another way to implement micro-jobs :-)

The queue would transparently handle the restart points.  Without the queue, 
you still need a way of organising the job order and state.

J