subunit-dev team mailing list archive

[Jelmer Vernooij <jelmer@xxxxxxxxx>]

On Sat, Jun 06, 2009 at 11:54:33AM +1000, Robert Collins wrote:
> On Fri, 2009-06-05 at 12:53 +0200, Jelmer Vernooij wrote:
> > > > > > >  * "skip-testsuite" command: Used to indicate that a particular
> > > > > > > testsuite (and all its tests) was skipped
> > > > Since testsuites include multiple tests, we'd like to be able to
> > > > report that we've skipped an entire testsuite and all the tests in it,
> > > > rather than reporting all the individual tests.
> > > I understand the use case, but not how you want it to work. I mean, the
> > > easiest way to skip an entire test suite is simply not to run it.

> > I'd like to be able to report that fact to the user.

> How about 
> if [ -z "$RUN_TDB" ]; then
>     tdb/suite | subunit-filter --add-prefix tdb
> else
>     echo test suite-tdb
>     echo skip suite-tdb
> fi

That doesn't imply afaik that all of the tests that start with suite-tdb are
skipped as well. Is there a defined hierarchy of tests in subunit?

> > > Oh. Yes, I think it doesn't make sense. In my C unit tests I make the
> > > individual binaries always return 0 if they complete, as subunit is
> > > doing the decision making about good/bad.
> > We'd still like to be able to use these tests outside of subunit, in
> > the same fashion that we did before and that was based on the exit
> > codes.
> Which is presumably why you want the raw output to be as clean as
> possible.
> Uhm. One thing you could do is to have a --subunit flag which turns on
> output code changing. I do wonder if its worth trying to keep raw output
> really clean though - with simple filters that will post process, is
> there really that much motivation to have the test scripts directly
> executable?
I don't want to force people to use subunit filters if they're
just working on a single testsuite. Most of these testsuites are also 
independent and don't share any code; I'd rather not add more complexity 
in each test suite (both because of the complexity itself and because of 
the effort required). 

One of the things I like about subunit is that it's pretty well
readable without a prettifier, is simple and doesn't require anything in the 
test runner other than following a few conventions about output. 
Subunit is not a different way of doing test results to me, it's something on 
top of what we already have that allows us to extract more information from
the test results.

So, for now it seems to make most sense to keep these extensions
specific to Samba. I've tried your suggestion about having a Subunit
filter that adds prefixes and that seems to work well so I'm happy
with the existing possibilities in the subunit protocol and the functionality 
in the subunit tools for now.

I now also have a diff-subunit command, written in Perl that I hope to
land in Samba as part of "make test" next week. I'll submit this
upstream as well, with a version of Subunit.pm that has all the
Samba-specific commands stripped out.

Cheers,

Jelmer

-- 
Jelmer Vernooij <jelmer@xxxxxxxxx> - http://samba.org/~jelmer/