testrepository-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On 17 July 2015 at 04:44, Jelmer Vernooij <jelmer@xxxxxxxxx> wrote:
> On Thu, Jul 16, 2015 at 07:14:17AM +1200, Robert Collins wrote:
>> Something I've wanted to do for a while is be able to report on 'this
>> test failed with CLANG, this with GCC' or 'test X fails on python 2.6,
>> 3.2'.
>>
>> We have an isolated test API - the instance_provision/dispose_execute functions.
>>
>> If we extend that by:
>> - adding a new variable $INSTANCE_PROFILE
>> - adding one or more ways to select profiles[1]
>> - namespacing test ids as they come in, and unwrapping the namespace
>> when we select them to run
>> - querying for test ids in all of the profiles[2]
>>
>> Then we should be able to do some group-by stuff in the UI, or via
>> direct subunit commands, easily enough.
> I like the general idea.
>
> This seems to have some overlap with Samba's environments feature, though
> it's not the same thing.

Can you point me at that ?

>> 1]: I'm thinking a list_profiles function in .testr.conf, which could
>> be defined as "echo py26 py27 py32 py34" (for a constant set) or could
>> call out to a helper that inspects whats available, or whatever.
>> Probably with a command line to: -p py27,py32. I think perhaps that
>> that should do a set intersection check though, and error on missing
>> ones.
> +1
>>
>> 2]: I think the pain of querying N times will be outweighed by the
>> interesting things users may do with partitioned profiles.
> Are there any examples you have in mind where test lists would differ significantly per
> profile?

Yeah - so here's a concrete example. In subunit we have:
 - shell tests
 - C tests
 - CPP tests
 - python tests

and we probably should be linting the codebase too.
 - pylint / pep8
 - splint
 - bash8

and the list of environments might be
py27 py27 py34 py35 py36 linters gcc clang bash zsh

where we'd run the python tests in the py environments, the linters
only in linters, the c and cpp in gcc and clang, and the shell tests
in bash and zsh

-Rob


-- 
Robert Collins <rbtcollins@xxxxxx>
Distinguished Technologist
HP Converged Cloud