yellow team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Wed, May 30, 2012 at 7:16 AM, Gary Poster <gary.poster@xxxxxxxxxxxxx> wrote:
> Hi Robert.  I asked Brad to go ahead and land his fix for bug 1004625,
> because I wanted to get the associated tests back in the parallel
> testing bandwagon while we hashed out what to do next.  Brad was
> reinstating previous behavior, which was and is (at least for now)
> desirable, using a small patch, so I approved it.

ok.

> This email is to start the process of hashing out next steps.
>
> You pointed out two bugs: 682772 and 682771.  These center around the
> fact that you don't want LP to generate duplicate test ids.
>
> Our thoughts are these, if I understand correctly.  I'm counting on
> other Yellow Squad folks clarifying my message if I go astray or leave
> something out.
>
> - The Python test runner and test tools have no problem with duplicate
> test names.  We've encountered this usage pattern on other projects as
> well.  We'd argue that a test tool that wants to fit in naturally within
> the Python testing world would be more convenient if it supported this
> existing pattern, of which Launchpad is but one example.

There may be a conceptual difference here. Duplicate test names -
shrug. Duplicate test ids - much more of a problem. I would say they
have a substantial problem when you consider running just one test - a
*very* common developer workflow. Launchpad is the only project I've
seen with duplicate test ids. Test ids, for clarity, default to the
fully qualified class name + . + the test method. This is
intrinsically unique unless test multiplication is used, and the only
two implementations I've run into are:
 - the one that doesn't assign unique ids that LP is using for some tests
 - the one that assigns unique ids via testtools' helper - in
testscenarios and bzr.

> - testr is the only component in our stack that has trouble with
> duplicate test names that we can see.  Even so, it almost does just fine
> with them.  As long as a testrunner expands a single test name to all
> associated tests, as ours did and now does again, the tests are run just
> fine, all consecutively on a single worker.  testr probably does not
> record the test time optimally (I'd guess it should aggregate?), and it
> does need a bit of work on test counting, but other than that seems to
> be fine, at least for roughshod usage.  I suspect that a bit of polish
> on those two items and a bit of warning documentation would make this a
> non-issue for testr users.
>
> In sum, we are not confident that those two bugs are in fact the right
> way to go.  We think testr should accommodate the test world, instead.
>
> Even if you disagree with us, we also don't think that addressing this
> one way or another is critical to the project--but as both the client
> and the TA, I suspect you might be able to have something to say about
> that if you wish. :-)

As the duplicates are doctests, and doctests are slow, I suspect this
has something to do with the 5 minute idle time you observed in high
CPU count parallelism. Just a WAG :).

You are right that from a minimal sense 'it will work', but it won't
be able to parallelise properly, nor can developers reliably find the
variant of a test to run from a backtrace etc.

Assigning unique ids is very easy to do using existing heavy lifting code.

-Rob