← Back to team overview

checkbox-dev team mailing list archive

Re: Update on the certification status meta-data / CEP-8


On Thu, Feb 5, 2015 at 1:42 PM, Zygmunt Krynicki
<zygmunt.krynicki@xxxxxxxxxxxxx> wrote:
> Hey.
> To refresh our usage of the mailing list I'd like to share the status
> of where we are with the meta-data story. The point of the story is to
> know the certification status of each job so that we can put it into
> reports.

Awesome, thanks for working on this! Later down the line we need to
think how to model this on our server-side tools/database.

> The specification on how that should look like from the outside has
> been merged to trunk earlier today. You can read it here [1].
> (side-story, we should build all CEPs into nice documents along with
> the rest of our documentation). The parts you may find especially
> interesting are all the examples that show how this looks like inside
> test plans and job units.

The examples are very clear and pretty cool. Will both syntaxes be
supported? At first glance I like:

apply blocker to job-1

since it's very natural-languagey, however, that screams "parser" and
a) we know parsers are painful, and b) a problem with natural language
is that people tend to not treat them very seriously, "ah, it's
natural language so it'll understand anything I throw at it" and we'll
start seeing "set job-1 to blocker" or other things. Strict validation
will be crucial here to help developers grok the new syntaxes.

The compact syntax is more strict in that respect which may be better
or worse. The bottom line is that there's no silver bullet when we
introduce new syntax and trying to validate and hand-hold developers
will be important.

> Over the past weeks I've made some progress into making that work:
>  - We now have JobDefinition.certification_status property and a
> corresponding SymbolDef with all the possible values. The validation
> code can capture simple cases of incorrect usage. That code has been
> in trunk for a while. It only got a minor lifting lately with the
> rename to 'certification-status'
>  - We also have the corresponding
> JobState.effective_certification_status. To refresh your memory, that
> is the class that holds mutable job state (e.g. results) as jobs
> themselves are immutable. This landed today. I'm super happy about how
> short and sweet that is thanks to POD and the custom field classes we
> have [2]. If you haven't seen that earlier I'd recommend reading the
> POD class we have and seeing how it can be used to simplify a lot of
> common coding patterns.

PODs are awesome, thanks!

>  - I've made some changes to the exporter base and to the xml and html
> classes so that we have this data and can surface it. I've just pushed
> that code here [3]. I haven't actually tried using it much so feedback
> is appreciated. I haven't spent any time on making the actual HTML
> report look nice. I'm sure it's possible but given how friendly
> (*cough*) XSLT is it'd rather do something else. Oh, tests pass as the
> canned xml/html reports now differ. I didn't bother updating them
> until we have an acceptable looking report.

How's progress on decoupling HTML from XML/XSLT? I guess this will
lessen the pain of doing what you describe, and I thought we'd decided
not to touch the XSLT and focus on the decoupling/templated native
HTML report instead.

> - I've started and I'm well into making all of the support machinery
> so that a TestPlanUnit class has the code that determines overrides
> efficiently. This is expanded in the subsequent point. Here I'm just
> focused on the raw parsers and computations required. It's all
> building on the xparsers module that was landed a while ago. This is
> long but relatively safe/well defined and easy to understand. I'll try
> to publish that code tomorrow.
>  - We have a branch with almost-but-not-quite-clean-history that moves
> test plans and whitelists closer together. The branch [4] is currently
> up for review. I'm looking for comments on the
> ProviderContent{Enumerator,Classifier,Loader} classes as well as hints
> on how the LazyFileContent class could be broken due to how naive my
> implementation is. Once that lands in one shape or another, I'd like
> to go over each use of the WhiteList class and replace it with similar
> TestPlan based code. Most of that will be easy. The hard parts will
> come later.
> - I had a look and I've started to code this but nothing's done yet at
> how we need to actually set JobState.effective_* fields. Currently
> that's an ad-hoc code slapped together here and there. That's
> obviously crap and I've created two new methods on a TestPlanUnit
> class to make that easier / better. One for bulk initialization and
> another one for job-by-job updates. The first method (I don't recall
> the name now and it's stashed away in my git tree) has to be called by
> all the users of a session whenever they are done with figuring out
> which test plan is being used. This can also be improved (so that
> there's only one such place) but that's another story. This bulk
> method combines all the overrides from all the sources and applies
> them to the state of any job it has access to. It's an evolution of
> the existing override handling code but it's both generic for any
> field and faster (much much faster). The second thing all the
> applications must do is to register a callback whenever a new job is
> added. Currently this is also ad-hoc but, again, there will be one
> method to call that does the right thing. It's also on a test plan
> unit class now and it's very similar to the bulk method but it
> optimizes in making small change efficient (as typically we call the
> callback quite a few times and each time just for one job. The
> apparent problem here is how that scales to multiple test plans. I
> don't know yet. It feels that this should all be handled by the
> session manager somehow but that would mean we need to teach the
> session manager about test plans, handle that in suspend/resume, etc.
> It's all possible but it's also somewhat complicated and I need to
> give it some more thought. Ideas and prototypes are much appreciated.
> So for a quick summary:
>  - spec done
>  - need to finish xparsers to handle new override syntax
>  - need to finish TestPlanUnit class to have bulk/single update
> methods to apply all overrides
>  - need to think about how to handle multiple test plans in that configuration
>  - need to think about how to handle test plan in suspend/resume code
> (we need to store the one's we're running so that after resume we can
> still apply overrides to generated jobs)
>  - real data for cdts missing
>  - real data for 14.04 prototyped [5], blocked by xparsers, can land
> after review once unblocked
> Links:
> [1] http://bazaar.launchpad.net/~checkbox-dev/checkbox/trunk/view/head:/cep/CEP-8.rst
> [2] http://bazaar.launchpad.net/~checkbox-dev/checkbox/trunk/revision/3565.1.3
> [3] https://code.launchpad.net/~zkrynicki/checkbox/cert-status-exporter
> [4] https://code.launchpad.net/~zkrynicki/checkbox/virtual-units/+merge/248699
> [5] http://pastebin.ubuntu.com/9882332/
> --
> Mailing list: https://launchpad.net/~checkbox-dev
> Post to     : checkbox-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~checkbox-dev
> More help   : https://help.launchpad.net/ListHelp

Follow ups