testrepository-dev team mailing list archive

[Jelmer Vernooij <jelmer@xxxxxxxxx>]

On Mon, Jul 27, 2015 at 09:33:00AM +1200, Robert Collins wrote:
> On 17 July 2015 at 04:44, Jelmer Vernooij <jelmer@xxxxxxxxx> wrote:
> > On Thu, Jul 16, 2015 at 07:14:17AM +1200, Robert Collins wrote:
> >> Something I've wanted to do for a while is be able to report on 'this
> >> test failed with CLANG, this with GCC' or 'test X fails on python 2.6,
> >> 3.2'.
> >>
> >> We have an isolated test API - the instance_provision/dispose_execute functions.
> >>
> >> If we extend that by:
> >> - adding a new variable $INSTANCE_PROFILE
> >> - adding one or more ways to select profiles[1]
> >> - namespacing test ids as they come in, and unwrapping the namespace
> >> when we select them to run
> >> - querying for test ids in all of the profiles[2]
> >>
> >> Then we should be able to do some group-by stuff in the UI, or via
> >> direct subunit commands, easily enough.
> > I like the general idea.
> >
> > This seems to have some overlap with Samba's environments feature, though
> > it's not the same thing.
> 
> Can you point me at that ?

They're the same thing we talked about earlier, but previously we talked about
the test result reporting side.

In Samba we have a bunch of client-side tests we want to run in different
(simulated) test environments. So we set up a specific set of servers
(i.e. simulating Windows Server 2003 with just a single DC), then run
a bunch of tests against those servers, then set up a new
set of servers (e.g. a replicated set of DCs), then run (mostly)
those same tests again. For most of these we just set up different
instances of Samba itself, but there is also a hack that allows us to
run against an instance of Windows in a VM.

While this is not exactly the same as what you're describing for profiles,
there are some similarities.

Jelmer