checkbox-dev team mailing list archive

[Sylvain Pineau <sylvain.pineau@xxxxxxxxxxxxx>]

I wanted to reply to the list.


-------- Forwarded Message --------
Subject: 	Re: [Checkbox-dev] Summary of the discussion about system states
Date: 	Thu, 23 Apr 2015 12:07:27 +0200
From: 	Sylvain Pineau <sylvain.pineau@xxxxxxxxxxxxx>
To: 	Zygmunt Krynicki <zygmunt.krynicki@xxxxxxxxxxxxx>



On 04/23/2015 11:42 AM, Zygmunt Krynicki wrote:
> Hi.
>
> This is a quick summary of the discussion about system states that we
> just had in a hangout. The core idea behind system states is to
> simplify job maintenance by modelling "states" and state transitions
> that affect the DUT.
>
> The classic example of this problem is the duplication (or
> quadruplication if you include the manual and automatic variant) of
> the before/after anything jobs. Those jobs express the requirement to
> test a particular component both before and after suspend that, as you
> know, often breaks something that works well on cold boot.
>
> The idea that Yung came up with works is composed of the following new
> bits of information.
>
> - A set of states, so perhaps (my interpretation) a new state unit
> - A new annotation on certain jobs that they cause a state transition
> (e.g. that the suspend job causes the system to go from
> not_seen_suspend to did_see_suspend state, hypothetically). Similarly
> the reboot job can cause the did_see_suspend state to reset to its
> nominal value, to express what is really happening with the DUT
> - A new annotation, either at a test plan level or at a job definition
> level, saying that we wish a job to run for all possible combinations
> of given states. In practice this can be a test for GPU that runs
> before and after suspend and before and after GPU switch.
> - A tier annotation on states that assists plainbox in ordering
> transitions. For example, we can say that we prefer to reboot first
> then suspend or vice versa. It doesn't change the semantics of
> anything but it does change the effective ordering of the sequence of
> state transition events.
>
> Having this information plainbox can synthesize a run list that
> contains additional jobs, derived from the base definition, and can
> ensure that all the required transitions occur and that all the tests
> are in the right spot in the list. Currently this is a very time
> consuming and fragile, manual process. This also means that we can
> drastically reduce the number of existing jobs by combining many of
> the before/after definitions into one definition.
It will also make the rerun feature quite complex as the system state at the
end of a testrun won't allow all jobs to be eligible for rerun according to
their state requirements, that's something to consider. Or we could imagine
that plainbox generates a list of jobs to get to that state)
> Lastly, we can take advantage of the existing part of the unit
> template system. All units can be parametric in plainbox. A template
> is simply a parametric unit with non-zero list of parameters. We can
> use that system to define non-template parametric units that have a
> parameter for each modelled system state.
>
> I hope this captures our discussions accurately. If not please post
> corrections for everyone to see. Let's keep discussing this on the
> mailing list.
Having states will effectively reduce the amount of jobs we maintain in both
job definitions units and testplan. But it will also hide the exact sequence
of tests only available in the run list internally. Without running the app
to get the selection screen it would be good also to have some sort of
plainbox dev command that will list all the jobs from a given testplan (with
a possible verbose mode detailing the state required/generated fro each
job).

> Thanks
> ZK