subunit-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jonathan Lange wrote:
> I've just updated Tribunal (https://launchpad.net/tribunal) to work
> with new versions of testtools.
> 
> In real life recently, Robert made comments about making Tribunal more
> subunit-oriented. I'm generally in favour of that, given the recent
> work that's been going on with subunit. However, I'm not exactly sure
> how it should work.
> 
> To interact with any test suite, Tribunal needs to be able to:
>   - discover tests
>   - run any arbitrary subset of them
>   - display the results
>
> Ideally, I'd like some kind of configuration or project file that you
> can give to Tribunal which tells it how to discover tests and how to
> run them. I think it would also be nice if Tribunal could display
> pretty much any arbitrary subunit stream, making it something of a
> results analysis tool, rather than just a runner.

That sounds very nice. subunit2gtk is an example of how that might work,
in terms of pipelines. I'd love to be able to delete it and point people
at a tribunal package :).

> The discovery part is pretty simple, I think. You just need to specify
> a command that lists all the tests.

Yes. It could just output a subunit stream and you can ignore the status
codes. Generator tests specifically really need to operate in roughly
this mode within python, to know what-exists.

> The run-arbitrary-subset bit is harder, I think, and I don't really
> know what to do there.

What I recommend folk do, is use the test ids as a way to specify
individual tests.

So, if in tribunal you want to run some set of tests FOO, you need a
command line that accepts (say) a ${test_ids} parameter.

list = "./bzr selftest --list-only"
run_all = "make check"
run_some = "make check TESTS=${test_ids}"

It might be nice, to also support bzr's load-list which can be a lot
faster and more precise:
run_some = "./bzr selftest --load-list ${test_ids_file}"
(and have tribunal write a list of ids, one per line before running the
command).

- -Rob
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkqjX1kACgkQ42zgmrPGrq7VEwCeMyxRDxu0ElDO1TkaCt6awlfe
SMIAn0b9qD+Xy0Z38sIMQlO2qusUsU4Y
=SSSu
-----END PGP SIGNATURE-----