subunit-dev team mailing list archive

[Jelmer Vernooij <jelmer@xxxxxxxxx>]

On Fri, Jun 05, 2009 at 10:04:03AM +1000, Robert Collins wrote:
> On Fri, 2009-06-05 at 09:54 +1000, Robert Collins wrote:
> > > Samba is getting pretty close to using "standard" subunit internally
> > > and I've separated the actual test runner from the displayer. I think
> > > I've got a pretty good idea now of what sort of extensions we'll need
> > > in Subunit.

> > > * a "testsuite" command that can be used to set a mandatory prefix for
> > > the test names
> Could you tell me a bit more about this.
This would be a part in the stream, which would start a particular
testsuite. In Samba, we have several different testsuites that are
pretty much ignorant of the overall layout. They each use simple test
names, but in the eventual test report we would like to have a
hierarchy, because it makes it easier to find where tests come from
but also because some of the simple tests names without the testsuite
name would overlap.

I'd like to achieve this by prepending the output of each
testsuite (which usually comes down to a single executable that spits
out subunit) with the prefix for the tests in that testsuite.

E.g. The actual output from the talloc testsuite executable contains:

...
test: lifeless
success: lifeless
test: free null
success: free null
...

Our overall test runner would do something like this (paraphrased):

echo "testsuite: talloc"
./talloc/testsuite
echo "testsuite: tdb"
./tdb/testsuite
echo "testsuite: ldb.ldap"
./ldb/testsuite ldap://localhost/
echo "testsuite: ldb.embedded"
./ldb/testsuite tdb://foo.tdb
echo "testsuite: pidl.foo"
./pidl/foo.pl | tap2subunit
...

(and another 500 more of these)

And subunit would interpret the tests being run as being named:

...
talloc.lifeless
talloc.free null
...

> And for the following, could you perhaps describe how these work/sample
> behaviour? I'm not clear if you're talking about library API's, perl
> binding specifics, or filters in a filter chain, or items in the stream.

> > >  * "skip-testsuite" command: Used to indicate that a particular
> > > testsuite (and all its tests) was skipped
Since testsuites include multiple tests, we'd like to be able to
report that we've skipped an entire testsuite and all the tests in it,
rather than reporting all the individual tests.

> > >  * "testsuite-count" command: Used to indicate the number of
> > > testsuites that's going to be run
We use this for progress indication, the output formatter counts the
number of testsuites run and the total and displays the percentage of
tests run so far. It's not necessarily a good indicator for speed or 
number of tests, as some testsuites may be significantly slower than others or
contain significantly more/less tests.

> > >  * "testsuite-result" command: Used to report the overall result of a
> > > testsuite (we have this to ensure that test commands use the right
> > > exit code depending on whether they reported failures).

> At least for this one specifically, subunit filters currently derive
> this from the behaviour across the stream; if you've specific rules
> about what counts as pass/fail we can look at doing that via options to
> control the heuristics, or a stream element as you suggest. I'm inclined
> to the former as it preserves the ability to aggregate.

We would derive the testsuite result from the string as well, but we'd
like to make sure the exit status of the testsuite command matches the
output of the command; if a command reported a single "fail" it should return 
non-zero as exit status. The last command probably doesn't really make
sense for subunit itself but just for Samba, as it's quite specific to
the way we do things.

Cheers,

Jelmer

Attachment: signature.asc
Description: Digital signature