checkbox-dev team mailing list archive

[Zygmunt Krynicki <zygmunt.krynicki@xxxxxxxxxxxxx>]

Hey.

To refresh our usage of the mailing list I'd like to share the status
of where we are with the meta-data story. The point of the story is to
know the certification status of each job so that we can put it into
reports.

The specification on how that should look like from the outside has
been merged to trunk earlier today. You can read it here [1].
(side-story, we should build all CEPs into nice documents along with
the rest of our documentation). The parts you may find especially
interesting are all the examples that show how this looks like inside
test plans and job units.

Over the past weeks I've made some progress into making that work:

 - We now have JobDefinition.certification_status property and a
corresponding SymbolDef with all the possible values. The validation
code can capture simple cases of incorrect usage. That code has been
in trunk for a while. It only got a minor lifting lately with the
rename to 'certification-status'

 - We also have the corresponding
JobState.effective_certification_status. To refresh your memory, that
is the class that holds mutable job state (e.g. results) as jobs
themselves are immutable. This landed today. I'm super happy about how
short and sweet that is thanks to POD and the custom field classes we
have [2]. If you haven't seen that earlier I'd recommend reading the
POD class we have and seeing how it can be used to simplify a lot of
common coding patterns.

 - I've made some changes to the exporter base and to the xml and html
classes so that we have this data and can surface it. I've just pushed
that code here [3]. I haven't actually tried using it much so feedback
is appreciated. I haven't spent any time on making the actual HTML
report look nice. I'm sure it's possible but given how friendly
(*cough*) XSLT is it'd rather do something else. Oh, tests pass as the
canned xml/html reports now differ. I didn't bother updating them
until we have an acceptable looking report.

- I've started and I'm well into making all of the support machinery
so that a TestPlanUnit class has the code that determines overrides
efficiently. This is expanded in the subsequent point. Here I'm just
focused on the raw parsers and computations required. It's all
building on the xparsers module that was landed a while ago. This is
long but relatively safe/well defined and easy to understand. I'll try
to publish that code tomorrow.

 - We have a branch with almost-but-not-quite-clean-history that moves
test plans and whitelists closer together. The branch [4] is currently
up for review. I'm looking for comments on the
ProviderContent{Enumerator,Classifier,Loader} classes as well as hints
on how the LazyFileContent class could be broken due to how naive my
implementation is. Once that lands in one shape or another, I'd like
to go over each use of the WhiteList class and replace it with similar
TestPlan based code. Most of that will be easy. The hard parts will
come later.

- I had a look and I've started to code this but nothing's done yet at
how we need to actually set JobState.effective_* fields. Currently
that's an ad-hoc code slapped together here and there. That's
obviously crap and I've created two new methods on a TestPlanUnit
class to make that easier / better. One for bulk initialization and
another one for job-by-job updates. The first method (I don't recall
the name now and it's stashed away in my git tree) has to be called by
all the users of a session whenever they are done with figuring out
which test plan is being used. This can also be improved (so that
there's only one such place) but that's another story. This bulk
method combines all the overrides from all the sources and applies
them to the state of any job it has access to. It's an evolution of
the existing override handling code but it's both generic for any
field and faster (much much faster). The second thing all the
applications must do is to register a callback whenever a new job is
added. Currently this is also ad-hoc but, again, there will be one
method to call that does the right thing. It's also on a test plan
unit class now and it's very similar to the bulk method but it
optimizes in making small change efficient (as typically we call the
callback quite a few times and each time just for one job. The
apparent problem here is how that scales to multiple test plans. I
don't know yet. It feels that this should all be handled by the
session manager somehow but that would mean we need to teach the
session manager about test plans, handle that in suspend/resume, etc.
It's all possible but it's also somewhat complicated and I need to
give it some more thought. Ideas and prototypes are much appreciated.

So for a quick summary:
 - spec done
 - need to finish xparsers to handle new override syntax
 - need to finish TestPlanUnit class to have bulk/single update
methods to apply all overrides
 - need to think about how to handle multiple test plans in that configuration
 - need to think about how to handle test plan in suspend/resume code
(we need to store the one's we're running so that after resume we can
still apply overrides to generated jobs)
 - real data for cdts missing
 - real data for 14.04 prototyped [5], blocked by xparsers, can land
after review once unblocked

Links:

[1] http://bazaar.launchpad.net/~checkbox-dev/checkbox/trunk/view/head:/cep/CEP-8.rst
[2] http://bazaar.launchpad.net/~checkbox-dev/checkbox/trunk/revision/3565.1.3
[3] https://code.launchpad.net/~zkrynicki/checkbox/cert-status-exporter
[4] https://code.launchpad.net/~zkrynicki/checkbox/virtual-units/+merge/248699
[5] http://pastebin.ubuntu.com/9882332/