testrepository-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On 27 July 2015 at 00:27, Jonathan Lange <jml@xxxxxxxxx> wrote:
> On Wed, 15 Jul 2015 at 21:14 Robert Collins <robertc@xxxxxxxxxxxxxxxxx>
> wrote:
>>
>> Something I've wanted to do for a while is be able to report on 'this
>> test failed with CLANG, this with GCC' or 'test X fails on python 2.6,
>> 3.2'.
>>
>
> Yes, this is something we want for flocker. Thanks for looking into this!
> (NB: We don't currently use testrepository, and some of my colleagues are
> considering writing their own test database).

I'd really love it if they contributed here :).

> More or less there's a cartesian product of small sets (e.g. {py26, py27,
> py32, pypy} x {ubuntu, redhat, centos} x {zfs-enabled, zfs-disabled} x
> {as-root x non-root}) and we want to know "this test failed on (py27,
> ubuntu, zfs-enabled, as-root)".

Yeah. So there's a few ways that could be modelled. One is to project
it to a single set:

py26/ubuntu/zfs/root
py26/ubuntu/zfs/-root
py26/ubuntu/-zfs/root
...

Another is to model all the sets and operators (x, +, -, =) to combine them.

pythons = 26 27
distros = ubuntu redhat
...
matrix = pythons x distros
knownbad = {py35} x {fedora}
matrix = matrix - knownbad

For this proposal I'm suggesting that testr accept the projection,
report based on the projection, and that another layer can experiment
with different ways to express the space.



>>
>> We have an isolated test API - the instance_provision/dispose_execute
>> functions.
>>
>> If we extend that by:
>> - adding a new variable $INSTANCE_PROFILE
>
>
> Can you give an example of what it might be set to?

https://github.com/rbtcollins/testrepository/blob/profiles/doc/MANUAL.txt#L289

"py27" would be an instance profile. Or - and I think we should
probably do this - "py27/ubuntu/zfs/root" - the opaque string
resulting from the projection.

>>
>> - adding one or more ways to select profiles[1]
>> - namespacing test ids as they come in, and unwrapping the namespace
>> when we select them to run
>
>
> As in, concatenate a string that uniquely identifies the profile with the
> test id?

yep. test foo.bar in profile py27/ubuntu/zfs/root would be
py27/ubuntu/zfs/root/py27.

>> - querying for test ids in all of the profiles[2]
>
>
> I don't quite get what this means.

I'm presuming that the set of tests that exists in a profile is not
constant. This is based on my experience with tox, where folk often
run different sets of tests: linters are one example, but another is
'only load tests X and Y because we're midway through porting to
python 3'.

>> Then we should be able to do some group-by stuff in the UI, or via
>> direct subunit commands, easily enough.
>
>
> Speaking of subunit, how would this feature integrate with the wire protocol
> and storage format?

The testid would be transformed on entry to testr, the storage format
isn't affected at all (except that the stored ids would be
namespaced).

The sql storage driver Matthew Treinish is working on probably would
want to store profile as a separate variable, so I expect some
iterations needed within the codebase to make that be preserved
cleanly down (though I'm not far enough through my prototype to really
say exactly how, yet).

>>
>>
>>
>> 1]: I'm thinking a list_profiles function in .testr.conf, which could
>> be defined as "echo py26 py27 py32 py34" (for a constant set) or could
>> call out to a helper that inspects whats available, or whatever.
>> Probably with a command line to: -p py27,py32. I think perhaps that
>> that should do a set intersection check though, and error on missing
>> ones.
>>
>
> That sounds reasonable.
>
> It doesn't sound like you're catering to the multi-dimensional "profile"
> case that I think we have. Is that right?

I'm not sure :) - certainly it wasn't top of my mind, but OTOH the
linters case certainly was, which is itself a small version of that.

-Rob


-- 
Robert Collins <rbtcollins@xxxxxx>
Distinguished Technologist
HP Converged Cloud