subunit-dev team mailing list archive

[Jelmer Vernooij <jelmer@xxxxxxxxx>]

On Fri, Jun 05, 2009 at 01:56:07PM +1000, Robert Collins wrote:
> On Fri, 2009-06-05 at 03:56 +0200, Jelmer Vernooij wrote:

> > This would be a part in the stream, which would start a particular
> > testsuite. In Samba, we have several different testsuites that are
> > pretty much ignorant of the overall layout. They each use simple test
> > names, but in the eventual test report we would like to have a
> > hierarchy, because it makes it easier to find where tests come from
> > but also because some of the simple tests names without the testsuite
> > name would overlap.

> > I'd like to achieve this by prepending the output of each
> > testsuite (which usually comes down to a single executable that spits
> > out subunit) with the prefix for the tests in that testsuite.

> > E.g. The actual output from the talloc testsuite executable contains:

> > ...
> > test: lifeless
> > success: lifeless
> > test: free null
> > success: free null
> > ...

> > Our overall test runner would do something like this (paraphrased):

> > echo "testsuite: talloc"
> > ./talloc/testsuite
> > echo "testsuite: tdb"
> > ./tdb/testsuite
> > echo "testsuite: ldb.ldap"
> > ./ldb/testsuite ldap://localhost/
> > echo "testsuite: ldb.embedded"
> > ./ldb/testsuite tdb://foo.tdb
> > echo "testsuite: pidl.foo"
> > ./pidl/foo.pl | tap2subunit
> > ...

> > (and another 500 more of these)

> > And subunit would interpret the tests being run as being named:

> > ...
> > talloc.lifeless
> > talloc.free null
> > ...

> You can do something similar with tags today. 

I never understood what the purpose of tags was; what are their
intended use? I don't see how they could modify the test name at
least.

> Another way you could do it is to add an option to subunit-filter. I
> think I'd rather have that than a stream command. Stream commands
> increase the complexity of every consumer of the stream; filters only
> increase the complexity of the specific filter code.

> ./talloc/testsuite | subunit-filter --add-prefix talloc
> ./tdb/testsuite | subunit-filter --add-prefix tdb
> ...

I'll play with this a bit. We'll probably still keep the testsuite
stuff around, but at least that way we won't need to extend subunit..

> > > And for the following, could you perhaps describe how these work/sample
> > > behaviour? I'm not clear if you're talking about library API's, perl
> > > binding specifics, or filters in a filter chain, or items in the stream.

> > > > >  * "skip-testsuite" command: Used to indicate that a particular
> > > > > testsuite (and all its tests) was skipped
> > Since testsuites include multiple tests, we'd like to be able to
> > report that we've skipped an entire testsuite and all the tests in it,
> > rather than reporting all the individual tests.
> I understand the use case, but not how you want it to work. I mean, the
> easiest way to skip an entire test suite is simply not to run it.
I'd like to be able to report that fact to the user.

> > > > >  * "testsuite-count" command: Used to indicate the number of
> > > > > testsuites that's going to be run

> > We use this for progress indication, the output formatter counts the
> > number of testsuites run and the total and displays the percentage of
> > tests run so far. It's not necessarily a good indicator for speed or 
> > number of tests, as some testsuites may be significantly slower than others or
> > contain significantly more/less tests.

> https://lists.launchpad.net/subunit-dev/msg00000.html may be a useful
> starting point for thinking about this.

> I agree we need to do something. But count up front is perhaps
> problematic with aggregating and filtering.
Yeah, this was why it was in the list with things we need for Samba
but might not want in Subunit. Samba already has this functionality at the
moment, and so we'll have to keep it around when moving towards
separate process subunit formatters.

> > > > >  * "testsuite-result" command: Used to report the overall result of a
> > > > > testsuite (we have this to ensure that test commands use the right
> > > > > exit code depending on whether they reported failures).

> > > At least for this one specifically, subunit filters currently derive
> > > this from the behaviour across the stream; if you've specific rules
> > > about what counts as pass/fail we can look at doing that via options to
> > > control the heuristics, or a stream element as you suggest. I'm inclined
> > > to the former as it preserves the ability to aggregate.

> > We would derive the testsuite result from the string as well, but we'd
> > like to make sure the exit status of the testsuite command matches the
> > output of the command; if a command reported a single "fail" it should return 
> > non-zero as exit status. The last command probably doesn't really make
> > sense for subunit itself but just for Samba, as it's quite specific to
> > the way we do things.

> Oh. Yes, I think it doesn't make sense. In my C unit tests I make the
> individual binaries always return 0 if they complete, as subunit is
> doing the decision making about good/bad.
We'd still like to be able to use these tests outside of subunit, in
the same fashion that we did before and that was based on the exit
codes.

Cheers,

Jelmer

Attachment: signature.asc
Description: Digital signature