testrepository-dev team mailing list archive

[Jonathan Lange <jml@xxxxxxxxx>]

On Sun, 26 Jul 2015 at 22:45 Robert Collins <robertc@xxxxxxxxxxxxxxxxx>
wrote:

> On 27 July 2015 at 00:27, Jonathan Lange <jml@xxxxxxxxx> wrote:
> > On Wed, 15 Jul 2015 at 21:14 Robert Collins <robertc@xxxxxxxxxxxxxxxxx>
> > wrote:
> >>
> >> Something I've wanted to do for a while is be able to report on 'this
> >> test failed with CLANG, this with GCC' or 'test X fails on python 2.6,
> >> 3.2'.
> >>
> >
> > Yes, this is something we want for flocker. Thanks for looking into this!
> > (NB: We don't currently use testrepository, and some of my colleagues are
> > considering writing their own test database).
>
> I'd really love it if they contributed here :).
>

Me too. I'll see what I can do.


> > More or less there's a cartesian product of small sets (e.g. {py26, py27,
> > py32, pypy} x {ubuntu, redhat, centos} x {zfs-enabled, zfs-disabled} x
> > {as-root x non-root}) and we want to know "this test failed on (py27,
> > ubuntu, zfs-enabled, as-root)".
>
> Yeah. So there's a few ways that could be modelled. One is to project
> it to a single set:
>
> py26/ubuntu/zfs/root
> py26/ubuntu/zfs/-root
> py26/ubuntu/-zfs/root
> ...
>
> Another is to model all the sets and operators (x, +, -, =) to combine
> them.
>
> pythons = 26 27
> distros = ubuntu redhat
> ...
> matrix = pythons x distros
> knownbad = {py35} x {fedora}
> matrix = matrix - knownbad
>
> For this proposal I'm suggesting that testr accept the projection,
> report based on the projection, and that another layer can experiment
> with different ways to express the space.
>
>
Hmm.

On the one hand, that's simple. On the other hand, it's hiding information
in a string, which never works out well, and raises compatibility concerns
for tools that do try to build information on top.


> >> - querying for test ids in all of the profiles[2]
> >
> >
> > I don't quite get what this means.
>
> I'm presuming that the set of tests that exists in a profile is not
> constant. This is based on my experience with tox, where folk often
> run different sets of tests: linters are one example, but another is
> 'only load tests X and Y because we're midway through porting to
> python 3'.
>
>
Oh yeah, that's definitely the case.

> Speaking of subunit, how would this feature integrate with the wire
protocol

> > and storage format?
>
> The testid would be transformed on entry to testr, the storage format
> isn't affected at all (except that the stored ids would be
> namespaced).
>
>
As in, the profile would become part of the testid?


> The sql storage driver Matthew Treinish is working on probably would
> want to store profile as a separate variable, so I expect some
> iterations needed within the codebase to make that be preserved
> cleanly down (though I'm not far enough through my prototype to really
> say exactly how, yet).
>
>
This is another example of why it would be useful to allow
multi-dimensional profiles: it would make querying an SQL database easier &
(probably) faster.

jml