subunit-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Fri, 2009-06-05 at 03:56 +0200, Jelmer Vernooij wrote:

> This would be a part in the stream, which would start a particular
> testsuite. In Samba, we have several different testsuites that are
> pretty much ignorant of the overall layout. They each use simple test
> names, but in the eventual test report we would like to have a
> hierarchy, because it makes it easier to find where tests come from
> but also because some of the simple tests names without the testsuite
> name would overlap.
> 
> I'd like to achieve this by prepending the output of each
> testsuite (which usually comes down to a single executable that spits
> out subunit) with the prefix for the tests in that testsuite.
> 
> E.g. The actual output from the talloc testsuite executable contains:
> 
> ...
> test: lifeless
> success: lifeless
> test: free null
> success: free null
> ...
> 
> Our overall test runner would do something like this (paraphrased):
> 
> echo "testsuite: talloc"
> ./talloc/testsuite
> echo "testsuite: tdb"
> ./tdb/testsuite
> echo "testsuite: ldb.ldap"
> ./ldb/testsuite ldap://localhost/
> echo "testsuite: ldb.embedded"
> ./ldb/testsuite tdb://foo.tdb
> echo "testsuite: pidl.foo"
> ./pidl/foo.pl | tap2subunit
> ...
> 
> (and another 500 more of these)
> 
> And subunit would interpret the tests being run as being named:
> 
> ...
> talloc.lifeless
> talloc.free null
> ...

You can do something similar with tags today. 

Another way you could do it is to add an option to subunit-filter. I
think I'd rather have that than a stream command. Stream commands
increase the complexity of every consumer of the stream; filters only
increase the complexity of the specific filter code.

./talloc/testsuite | subunit-filter --add-prefix talloc
./tdb/testsuite | subunit-filter --add-prefix tdb
...

> > And for the following, could you perhaps describe how these work/sample
> > behaviour? I'm not clear if you're talking about library API's, perl
> > binding specifics, or filters in a filter chain, or items in the stream.
> 
> > > >  * "skip-testsuite" command: Used to indicate that a particular
> > > > testsuite (and all its tests) was skipped
> Since testsuites include multiple tests, we'd like to be able to
> report that we've skipped an entire testsuite and all the tests in it,
> rather than reporting all the individual tests.

I understand the use case, but not how you want it to work. I mean, the
easiest way to skip an entire test suite is simply not to run it.

> > > >  * "testsuite-count" command: Used to indicate the number of
> > > > testsuites that's going to be run

> We use this for progress indication, the output formatter counts the
> number of testsuites run and the total and displays the percentage of
> tests run so far. It's not necessarily a good indicator for speed or 
> number of tests, as some testsuites may be significantly slower than others or
> contain significantly more/less tests.

https://lists.launchpad.net/subunit-dev/msg00000.html may be a useful
starting point for thinking about this.

I agree we need to do something. But count up front is perhaps
problematic with aggregating and filtering.

> > > >  * "testsuite-result" command: Used to report the overall result of a
> > > > testsuite (we have this to ensure that test commands use the right
> > > > exit code depending on whether they reported failures).
> 
> > At least for this one specifically, subunit filters currently derive
> > this from the behaviour across the stream; if you've specific rules
> > about what counts as pass/fail we can look at doing that via options to
> > control the heuristics, or a stream element as you suggest. I'm inclined
> > to the former as it preserves the ability to aggregate.
> 
> We would derive the testsuite result from the string as well, but we'd
> like to make sure the exit status of the testsuite command matches the
> output of the command; if a command reported a single "fail" it should return 
> non-zero as exit status. The last command probably doesn't really make
> sense for subunit itself but just for Samba, as it's quite specific to
> the way we do things.

Oh. Yes, I think it doesn't make sense. In my C unit tests I make the
individual binaries always return 0 if they complete, as subunit is
doing the decision making about good/bad.

-Rob

Attachment: signature.asc
Description: This is a digitally signed message part