yellow team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Thu, Nov 24, 2011 at 10:59 AM, Benji York <benji.york@xxxxxxxxxxxxx> wrote:
>> Our test distribution per layer is not very even - I highly doubt that
>> we'd be able to meet a reduction to 15% of the current time splitting
>> per layer.
>
> Let's look at the test distribution: The last buildbot run took 360
> minutes.  There were 4 layers that took longer than 11 minutes to run:
> 55, 56, 65, and 99 minutes.  All the other layers add up to about
> 60 minutes.

So the shortest run -j could give is 99 minutes, or 27% runtime. I
don't see how you can bisect a layer, unless you mean 'create a fake
layer extending it and manually allocate 50% of the tests to it'. That
seems like a non-starter to me - way to much maintenance overhead.

> If we bisect the four largest layers (to make it so the test runner's
> blind layer scheduling can't bite us too hard) and assume that running 4
> layers simultaneously imposes no more than a 50% overhead, then we would
> be right at 40% of the current running time.
>
> Reasoning sidebar: 99 is the length in minutes of the longest layer; it
> was bisected, but even then its other half is still the longest
> remaining layer so for pessimism's sake we assume they get run one after
> another.  All the other layers would be finished by then, so that gives
> us 99*1.50/360 = .41.
>
> Even if we assume no parallelization overhead, per-test scheduling (as
> opposed to per-layer as above) and four-way parallelization, we'll still
> be at 25% of the original time, so I'm interested in ideas as to how we
> might achieve a reduction to 15% of the original time.

If local parallelisation will work, testr run --parallel will load
balance all the tests optimally based on previous performance - a
single run from e.g. ec2 can tell us which tests are slow and let it
decide from there.

>> The other issue of shared global state that will bite us,
>> will also be a significant issue with -j, unless a remoting facility
>> is brought in (and at that point it seems to be reinventing
>> subunit.... :P).
>
> This is the real catch.  If the tests haven't been written to be
> parallelizable (which LP's certainly have not), then global state
> collisions accumulated over years of assuming non-parallel tests could
> be hard to fix.  On the other hand, if fixing them turns out to be easy,
> then using the test runner's built-in parallelization (-j) would be the
> most bang for the buck.

bin/test --parallel already exists and does better splitting than -j,
so I disagree that -j would be the best approach, *if* the collisions
etc are easy to fix :).

-Rob