subunit-dev team mailing list archive

[Jonathan Lange <jml@xxxxxxxxx>]

On Sun, Sep 6, 2009 at 5:06 PM, Robert Collins
<robertc@xxxxxxxxxxxxxxxxx> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jonathan Lange wrote:
>> I've just updated Tribunal (https://launchpad.net/tribunal) to work
>> with new versions of testtools.
>>
>> In real life recently, Robert made comments about making Tribunal more
>> subunit-oriented. I'm generally in favour of that, given the recent
>> work that's been going on with subunit. However, I'm not exactly sure
>> how it should work.
>>
>> To interact with any test suite, Tribunal needs to be able to:
>>   - discover tests
>>   - run any arbitrary subset of them
>>   - display the results
>>
>> Ideally, I'd like some kind of configuration or project file that you
>> can give to Tribunal which tells it how to discover tests and how to
>> run them. I think it would also be nice if Tribunal could display
>> pretty much any arbitrary subunit stream, making it something of a
>> results analysis tool, rather than just a runner.
>
> That sounds very nice. subunit2gtk is an example of how that might work,
> in terms of pipelines. I'd love to be able to delete it and point people
> at a tribunal package :).
>

I'll see what I can do.

>> The discovery part is pretty simple, I think. You just need to specify
>> a command that lists all the tests.
>
> Yes. It could just output a subunit stream and you can ignore the status
> codes. Generator tests specifically really need to operate in roughly
> this mode within python, to know what-exists.
>

Makes sense.

>> The run-arbitrary-subset bit is harder, I think, and I don't really
>> know what to do there.
>
> What I recommend folk do, is use the test ids as a way to specify
> individual tests.
>
> So, if in tribunal you want to run some set of tests FOO, you need a
> command line that accepts (say) a ${test_ids} parameter.
>
> list = "./bzr selftest --list-only"
> run_all = "make check"
> run_some = "make check TESTS=${test_ids}"
>
> It might be nice, to also support bzr's load-list which can be a lot
> faster and more precise:
> run_some = "./bzr selftest --load-list ${test_ids_file}"
> (and have tribunal write a list of ids, one per line before running the
> command).
>

I think that's a good idea. It might also be nice for tools to read
the list of tests from stdin, but that's a detail.

Thanks,
jml