launchpad-dev team mailing list archive

[Julian Edwards <julian.edwards@xxxxxxxxxxxxx>]

On Monday 23 August 2010 15:34:04 Aaron Bentley wrote:
> On 08/12/2010 10:04 AM, Julian Edwards wrote:
> > On Wednesday 11 August 2010 23:09:46 Michael Hudson wrote:
> >> On Wed, 11 Aug 2010 16:34:27 -0500, Edwin Grubbs
> > 
> > <edwin.grubbs@xxxxxxxxxxxxx> wrote:
> >>> I think this shows that if we run everything through a queue, there
> >>> needs to be a way to run really quick tasks in parallel with really
> >>> big tasks. You wouldn't want a single POFile export for a project to
> >>> be stuck behind the processing of 75k files for all of ubuntu. There
> >>> could be a slow lane and a fast lane. If the jobs have a method for
> >>> estimating the length of time reasonably well, we could also have a
> >>> super-fast lane, where the page decides not to queue the task at all.
> >> 
> >> https://dev.launchpad.net/CodeTeam/ExpressLane is an idea of Aaron's in
> >> this direction.
> > 
> > That seems awfully similar to the build farm's scoring mechanism.
> 
> I really don't think the express lane is similar to the build farm's
> scoring mechanism.  
> The express lane does not require the use of scores.

That's like saying a teapot doesn't use a windscreen wiper.

ie. The scoring is an implementation detail.

>  And its purpose is to ensure that tasks which can complete quickly have
> low latency, which is not a characteristic that the build farm currently
> has.

I have discussed this to death with many people in Canonical now and that's a 
popular misconception.

There's a tradeoff between utilisation and latency.  If you keep a resource 
open for high-priority jobs requiring no latency then you're wasting that 
resource.  And once you dispatch something to that resource, and another high-
priority job comes along, what do you do with it?

Given nR resources, one will be becoming free within X seconds at any point in 
time.  X is inversely proportional to nR, so we can mostly nullify the wait by 
adding resources.  We also make the best overall use of the resources, which 
in itself reduces latency for everything.

You also have to ask yourself if a high-priority job really has to dispatch 
*now*, or can it wait X seconds.  I bet most of the time it can wait, and this 
the strategy I have with the high-priority package builds (security, private, 
OEM etc)

Cheers
J