launchpad-dev team mailing list archive

[Jonathan Lange <jml@xxxxxxxxxxxxx>]

On Wed, Oct 6, 2010 at 9:49 AM, Michael Nelson
<michael.nelson@xxxxxxxxxxxxx> wrote:
> I just caused a test-fix using ec2 land with a windmill test that
> *did* fail, but the run was reported as successful (details below).
>
> I noticed a thread back in June with the same subject, but am not sure
> if it was resolved and this is a regression, or it was never
> resolved...
>

I never heard that it was resolved.

> ---------- Forwarded message ----------
> From: Michael Nelson <michael.nelson@xxxxxxxxxxxxx>
> Date: Tue, Oct 5, 2010 at 9:13 PM
> Subject: Test results: 652838-select-diffs-for-syncing => db-devel: SUCCESS
> To: michael.nelson@xxxxxxxxxxxxx
>
>
> Tests started at approximately 2010-10-05 15:27:08.255466
> Source: bzr+ssh://bazaar.launchpad.net/~michael.nelson/launchpad/652838-select-diffs-for-syncing
> r9844
> Target: bzr+ssh://bazaar.launchpad.net/~launchpad-pqm/launchpad/db-devel/ r9862
>
> 3745 tests run in 3:45:38.815244, 0 failures, 0 errors
>

This is the key bit. There are normally 10k+ tests, but the test
runner has only detected ~4k. That means that a whole bunch of tests
are being ignored.

If you look in the attachment, you'll see there's a
LayerIsolationError, and that every single failure or error after that
is ignored.  You can verify this by downloading the full output and
using tools like "subunit-filter" or "testr load" to see it.

Basically, subunit sees a "test:" command without a matching result
command (e.g. "success:", "failure:") and then proceeds to assume that
all of the rest of the input it gets is just regular stdout/stderr
dumpage.

That means there are two problems here:
  1. zope.testrunner is generating invalid subunit output in the case
of catastrophic layer failure.
  2. subunit is not robust in the case of invalid input

http://paste.ubuntu.com/507163/ shows a simplified stream that
reproduces the problem, and I've uploaded
https://code.edge.launchpad.net/~jml/subunit/masked-errors/+merge/37725
for the consideration of the other subunit hackers.

I'll try to figure something out with zope.testrunner now. I'll report
back to the list when I've made progress or given up.

jml