testrepository-dev team mailing list archive

[Jelmer Vernooij <jelmer@xxxxxxxxx>]

On Thu, Jul 16, 2015 at 07:14:17AM +1200, Robert Collins wrote:
> Something I've wanted to do for a while is be able to report on 'this
> test failed with CLANG, this with GCC' or 'test X fails on python 2.6,
> 3.2'.
> 
> We have an isolated test API - the instance_provision/dispose_execute functions.
> 
> If we extend that by:
> - adding a new variable $INSTANCE_PROFILE
> - adding one or more ways to select profiles[1]
> - namespacing test ids as they come in, and unwrapping the namespace
> when we select them to run
> - querying for test ids in all of the profiles[2]
> 
> Then we should be able to do some group-by stuff in the UI, or via
> direct subunit commands, easily enough.
I like the general idea.

This seems to have some overlap with Samba's environments feature, though
it's not the same thing.

> 1]: I'm thinking a list_profiles function in .testr.conf, which could
> be defined as "echo py26 py27 py32 py34" (for a constant set) or could
> call out to a helper that inspects whats available, or whatever.
> Probably with a command line to: -p py27,py32. I think perhaps that
> that should do a set intersection check though, and error on missing
> ones.
+1
> 
> 2]: I think the pain of querying N times will be outweighed by the
> interesting things users may do with partitioned profiles.
Are there any examples you have in mind where test lists would differ significantly per
profile?

Cheers,

Jelmer