testrepository-dev team mailing list archive

[Jelmer Vernooij <jelmer@xxxxxxxxx>]

On 30/07/15 11:59, Robert Collins wrote:
> On 27 July 2015 at 14:45, Jelmer Vernooij <jelmer@xxxxxxxxx> wrote:
>> On Mon, Jul 27, 2015 at 09:33:00AM +1200, Robert Collins wrote:
>>> On 17 July 2015 at 04:44, Jelmer Vernooij <jelmer@xxxxxxxxx> wrote:
>>>> On Thu, Jul 16, 2015 at 07:14:17AM +1200, Robert Collins wrote:
>>>>> Something I've wanted to do for a while is be able to report on 'this
>>>>> test failed with CLANG, this with GCC' or 'test X fails on python 2.6,
>>>>> 3.2'.
>>>>>
>>>>> We have an isolated test API - the instance_provision/dispose_execute functions.
>>>>>
>>>>> If we extend that by:
>>>>> - adding a new variable $INSTANCE_PROFILE
>>>>> - adding one or more ways to select profiles[1]
>>>>> - namespacing test ids as they come in, and unwrapping the namespace
>>>>> when we select them to run
>>>>> - querying for test ids in all of the profiles[2]
>>>>>
>>>>> Then we should be able to do some group-by stuff in the UI, or via
>>>>> direct subunit commands, easily enough.
>>>> I like the general idea.
>>>>
>>>> This seems to have some overlap with Samba's environments feature, though
>>>> it's not the same thing.
>>>
>>> Can you point me at that ?
>>
>> They're the same thing we talked about earlier, but previously we talked about
>> the test result reporting side.
>>
>> In Samba we have a bunch of client-side tests we want to run in different
>> (simulated) test environments. So we set up a specific set of servers
>> (i.e. simulating Windows Server 2003 with just a single DC), then run
>> a bunch of tests against those servers, then set up a new
>> set of servers (e.g. a replicated set of DCs), then run (mostly)
>> those same tests again. For most of these we just set up different
>> instances of Samba itself, but there is also a hack that allows us to
>> run against an instance of Windows in a VM.
>>
>> While this is not exactly the same as what you're describing for profiles,
>> there are some similarities.
> 
> So it seems like a subset of profiles. How is it different?

The test runner takes care of the environments, not the test tool (like
e.g. testr). The profile is also encoded in the test name when reporting
results.

Each environment has some setup and teardown code associated with it.
In particular, our test runner knows how to reuse existing servers when
setting up new environments.

Cheers,

Jelmer