subunit-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Fri, 2009-06-05 at 12:53 +0200, Jelmer Vernooij wrote:

> > You can do something similar with tags today. 
> 
> I never understood what the purpose of tags was; what are their
> intended use? I don't see how they could modify the test name at
> least.

Well they can't change the name, but you could tag all those tests. That
would let you tell which particulars suite test-foo was failing.

> > > > And for the following, could you perhaps describe how these work/sample
> > > > behaviour? I'm not clear if you're talking about library API's, perl
> > > > binding specifics, or filters in a filter chain, or items in the stream.
> 
> > > > > >  * "skip-testsuite" command: Used to indicate that a particular
> > > > > > testsuite (and all its tests) was skipped
> > > Since testsuites include multiple tests, we'd like to be able to
> > > report that we've skipped an entire testsuite and all the tests in it,
> > > rather than reporting all the individual tests.
> > I understand the use case, but not how you want it to work. I mean, the
> > easiest way to skip an entire test suite is simply not to run it.
>
> I'd like to be able to report that fact to the user.

How about 
if [ -z "$RUN_TDB" ]; then
    tdb/suite | subunit-filter --add-prefix tdb
else
    echo test suite-tdb
    echo skip suite-tdb
fi

> > > > > >  * "testsuite-count" command: Used to indicate the number of
> > > > > > testsuites that's going to be run
> 
> > > We use this for progress indication, the output formatter counts the
> > > number of testsuites run and the total and displays the percentage of
> > > tests run so far. It's not necessarily a good indicator for speed or 
> > > number of tests, as some testsuites may be significantly slower than others or
> > > contain significantly more/less tests.
> 
> > https://lists.launchpad.net/subunit-dev/msg00000.html may be a useful
> > starting point for thinking about this.
> 
> > I agree we need to do something. But count up front is perhaps
> > problematic with aggregating and filtering.
>
> Yeah, this was why it was in the list with things we need for Samba
> but might not want in Subunit. Samba already has this functionality at the
> moment, and so we'll have to keep it around when moving towards
> separate process subunit formatters.

I'm keen to get a good answer in subunit. The things I want to permit
are: stream filtering, aggregation and post processing. The things I
want to enable are test runs that report the total tests to be run up
front, or that report bits as they know about them. Essentially
reporters should be able to take advantage of as much information as a
given test run has.

> > Oh. Yes, I think it doesn't make sense. In my C unit tests I make the
> > individual binaries always return 0 if they complete, as subunit is
> > doing the decision making about good/bad.
> We'd still like to be able to use these tests outside of subunit, in
> the same fashion that we did before and that was based on the exit
> codes.

Which is presumably why you want the raw output to be as clean as
possible.
Uhm. One thing you could do is to have a --subunit flag which turns on
output code changing. I do wonder if its worth trying to keep raw output
really clean though - with simple filters that will post process, is
there really that much motivation to have the test scripts directly
executable?

-Rob

Attachment: signature.asc
Description: This is a digitally signed message part