yellow team mailing list archive

[Gary Poster <gary.poster@xxxxxxxxxxxxx>]

On 05/29/2012 11:36 PM, Robert Collins wrote:
> On Wed, May 30, 2012 at 7:16 AM, Gary Poster <gary.poster@xxxxxxxxxxxxx> wrote:
>> Hi Robert.  I asked Brad to go ahead and land his fix for bug 1004625,
>> because I wanted to get the associated tests back in the parallel
>> testing bandwagon while we hashed out what to do next.  Brad was
>> reinstating previous behavior, which was and is (at least for now)
>> desirable, using a small patch, so I approved it.
> 
> ok.
> 
>> This email is to start the process of hashing out next steps.
>>
>> You pointed out two bugs: 682772 and 682771.  These center around the
>> fact that you don't want LP to generate duplicate test ids.
>>
>> Our thoughts are these, if I understand correctly.  I'm counting on
>> other Yellow Squad folks clarifying my message if I go astray or leave
>> something out.
>>
>> - The Python test runner and test tools have no problem with duplicate
>> test names.  We've encountered this usage pattern on other projects as
>> well.  We'd argue that a test tool that wants to fit in naturally within
>> the Python testing world would be more convenient if it supported this
>> existing pattern, of which Launchpad is but one example.
> 
> There may be a conceptual difference here. Duplicate test names -
> shrug. Duplicate test ids - much more of a problem. I would say they
> have a substantial problem when you consider running just one test - a
> *very* common developer workflow. Launchpad is the only project I've
> seen with duplicate test ids. Test ids, for clarity, default to the
> fully qualified class name + . + the test method. 

Yes, I was using "name" to mean "fully qualified name" or your
description of "id".  Thanks for the clarity.

> This is
> intrinsically unique unless test multiplication is used, and the only
> two implementations I've run into are:
>  - the one that doesn't assign unique ids that LP is using for some tests
>  - the one that assigns unique ids via testtools' helper - in
> testscenarios and bzr.

Right.  We've seen it elsewhere, as I said.

> 
>> - testr is the only component in our stack that has trouble with
>> duplicate test names that we can see.  Even so, it almost does just fine
>> with them.  As long as a testrunner expands a single test name to all
>> associated tests, as ours did and now does again, the tests are run just
>> fine, all consecutively on a single worker.  testr probably does not
>> record the test time optimally (I'd guess it should aggregate?), and it
>> does need a bit of work on test counting, but other than that seems to
>> be fine, at least for roughshod usage.  I suspect that a bit of polish
>> on those two items and a bit of warning documentation would make this a
>> non-issue for testr users.
>>
>> In sum, we are not confident that those two bugs are in fact the right
>> way to go.  We think testr should accommodate the test world, instead.
>>
>> Even if you disagree with us, we also don't think that addressing this
>> one way or another is critical to the project--but as both the client
>> and the TA, I suspect you might be able to have something to say about
>> that if you wish. :-)
> 
> As the duplicates are doctests, and doctests are slow, I suspect this
> has something to do with the 5 minute idle time you observed in high
> CPU count parallelism. Just a WAG :).

Definitely possible!  Good thought.

> You are right that from a minimal sense 'it will work', but it won't
> be able to parallelise properly, 

...for some medium-high bar of "properly"; and given that testr *could*
aggregate times and handle the test counts, as we mentioned, to make
this largely a non-issue...

> nor can developers reliably find the
> variant of a test to run from a backtrace etc.

Granted.

> Assigning unique ids is very easy to do using existing heavy lifting code.

Cool.  I'm marking it as a low priority task for us.  It's also
reasonable for a bug squad to tackle from the developer perspective.
When we switch to bug squad time we could also look at it then.

Thanks,

Gary