launchpad-dev team mailing list archive

[Brad Crittenden <bac@xxxxxxxxxxxxx>]

Hi,

This email is mainly directed to the Yellow Squad, since we're fighting lots of test isolation problems right now due to our work with parallelizing our test suite.  But it may be interesting to others.

Yesterday I landed a fix for bug 989208 which makes '/bin/test --load-list testfile' run the tests in the same order as listed in the input file.  When the tests are run they are grouped by layer, so the guarantee now is that the ordering by layer is preserved.  Of course, the test list presented is probably the result of a previous run so they will already be grouped by layer.

This change helps in figuring out test isolation errors.  The negatively interacting tests are usually all in the same layer, so the tests from other layers can be ignored.  Also, tests that occur after the failed test can be ignored since they did not contribute to the error.  To determine the layer in which the error occurred you can use --load-list with --list-tests to see the layer breaks.

Steps:

1) Create a test list (one test per line) to be fed to --load-list from your failed test run. 

Let's assume you save the file as worker0.txt.

2) Find the layer where the bad test is run and the first test in that layer:

bin/test --load-list worker0.txt --list-tests > layered.txt

3) In layered.text find the test of interest and then search backwards for the layer marker.  Note the first test in that layer.  In worker-0.txt, copy and paste the set of tests starting with the first in the layer to the bad test and save as worker0-trimmed.txt.  (This change has to be made the original file, not the one produced by --list-tests, as it names some tests in a way that is not compatible with --load-list.)

4) Now you can run just the tests that are candidates for causing the isolation error:

/bin/test -vv --load-list worker0-trimmed.txt

These instructions will be added to https://dev.launchpad.net/ParallelTests/ResultsLog. 

--Brad