yellow team mailing list archive
-
yellow team
-
Mailing list archive
-
Message #00893
Re: Duplicate test names
On Wed, May 30, 2012 at 7:16 AM, Gary Poster <gary.poster@xxxxxxxxxxxxx> wrote:
> Hi Robert. I asked Brad to go ahead and land his fix for bug 1004625,
> because I wanted to get the associated tests back in the parallel
> testing bandwagon while we hashed out what to do next. Brad was
> reinstating previous behavior, which was and is (at least for now)
> desirable, using a small patch, so I approved it.
ok.
> This email is to start the process of hashing out next steps.
>
> You pointed out two bugs: 682772 and 682771. These center around the
> fact that you don't want LP to generate duplicate test ids.
>
> Our thoughts are these, if I understand correctly. I'm counting on
> other Yellow Squad folks clarifying my message if I go astray or leave
> something out.
>
> - The Python test runner and test tools have no problem with duplicate
> test names. We've encountered this usage pattern on other projects as
> well. We'd argue that a test tool that wants to fit in naturally within
> the Python testing world would be more convenient if it supported this
> existing pattern, of which Launchpad is but one example.
There may be a conceptual difference here. Duplicate test names -
shrug. Duplicate test ids - much more of a problem. I would say they
have a substantial problem when you consider running just one test - a
*very* common developer workflow. Launchpad is the only project I've
seen with duplicate test ids. Test ids, for clarity, default to the
fully qualified class name + . + the test method. This is
intrinsically unique unless test multiplication is used, and the only
two implementations I've run into are:
- the one that doesn't assign unique ids that LP is using for some tests
- the one that assigns unique ids via testtools' helper - in
testscenarios and bzr.
> - testr is the only component in our stack that has trouble with
> duplicate test names that we can see. Even so, it almost does just fine
> with them. As long as a testrunner expands a single test name to all
> associated tests, as ours did and now does again, the tests are run just
> fine, all consecutively on a single worker. testr probably does not
> record the test time optimally (I'd guess it should aggregate?), and it
> does need a bit of work on test counting, but other than that seems to
> be fine, at least for roughshod usage. I suspect that a bit of polish
> on those two items and a bit of warning documentation would make this a
> non-issue for testr users.
>
> In sum, we are not confident that those two bugs are in fact the right
> way to go. We think testr should accommodate the test world, instead.
>
> Even if you disagree with us, we also don't think that addressing this
> one way or another is critical to the project--but as both the client
> and the TA, I suspect you might be able to have something to say about
> that if you wish. :-)
As the duplicates are doctests, and doctests are slow, I suspect this
has something to do with the 5 minute idle time you observed in high
CPU count parallelism. Just a WAG :).
You are right that from a minimal sense 'it will work', but it won't
be able to parallelise properly, nor can developers reliably find the
variant of a test to run from a backtrace etc.
Assigning unique ids is very easy to do using existing heavy lifting code.
-Rob
Follow ups
References