testrepository-dev team mailing list archive

[Jonathan Lange <jml@xxxxxxxxx>]

On Wed, 15 Jul 2015 at 21:14 Robert Collins <robertc@xxxxxxxxxxxxxxxxx>
wrote:

> Something I've wanted to do for a while is be able to report on 'this
> test failed with CLANG, this with GCC' or 'test X fails on python 2.6,
> 3.2'.
>
>
Yes, this is something we want for flocker. Thanks for looking into this!
(NB: We don't currently use testrepository, and some of my colleagues are
considering writing their own test database).

More or less there's a cartesian product of small sets (e.g. {py26, py27,
py32, pypy} x {ubuntu, redhat, centos} x {zfs-enabled, zfs-disabled} x
{as-root x non-root}) and we want to know "this test failed on (py27,
ubuntu, zfs-enabled, as-root)".


> We have an isolated test API - the instance_provision/dispose_execute
> functions.
>
If we extend that by:
> - adding a new variable $INSTANCE_PROFILE
>

Can you give an example of what it might be set to?


> - adding one or more ways to select profiles[1]
> - namespacing test ids as they come in, and unwrapping the namespace
> when we select them to run
>

As in, concatenate a string that uniquely identifies the profile with the
test id?


> - querying for test ids in all of the profiles[2]
>

I don't quite get what this means.


> Then we should be able to do some group-by stuff in the UI, or via
> direct subunit commands, easily enough.


Speaking of subunit, how would this feature integrate with the wire
protocol and storage format?


>
>
1]: I'm thinking a list_profiles function in .testr.conf, which could
> be defined as "echo py26 py27 py32 py34" (for a constant set) or could
> call out to a helper that inspects whats available, or whatever.
> Probably with a command line to: -p py27,py32. I think perhaps that
> that should do a set intersection check though, and error on missing
> ones.
>
>
That sounds reasonable.

It doesn't sound like you're catering to the multi-dimensional "profile"
case that I think we have. Is that right?

cheers,
jml