checkbox-dev team mailing list archive

[Zygmunt Krynicki <zygmunt.krynicki@xxxxxxxxxxxxx>]

Hey

I wanted keep everyone in the loop about thunderbolt and about the
idea to use manifests:

12:52 <@zyga> spineau: let's talk about manifests
12:52 <@spineau> zyga: to handle job upgrade?
12:53 <@zyga> spineau: in my earlier drafts I was proposing to add a
manifest entry unit. each item would have an ID, a name (as listed in
the manifest), a type (bool, int, we can add more as needed) and
optional unit type
12:53 <@zyga> spineau: no, for thunderbolt
12:53 <@zyga> spineau: so for TB, we could say that there's a unit like this:
12:53 <@zyga> unit: manifest entry
12:53 <@zyga> id: number-of-thunderbolt-ports
12:53 <@zyga> type: int
12:54 <@zyga> _name: Number of thunderbolt ports
12:54 <@zyga> _description: This unit describes the number of external
thunderbolt ports as visible from the POV of the user.
12:54 <@zyga> (I would now rename the id to
number-of-external-thunderbolt-ports)
12:54 <@zyga> that's it
12:55 <@zyga> plainbox would see that, would load a manifest file and
expose it as a special resource like thing
12:55 <@zyga> spineau: in jobs we could say that a job requires:
number-of-thunderbolt-ports > 0
12:55 <@zyga> (now I would use underscore instead of dashes)
12:56 <@zyga> spineau: and this could be a special requirement on
manifes entry or a normal requirement, not sure yet
12:56 <@zyga> spineau: the actual manifest file could be rfc822 or json
12:56 <@zyga> spineau: we could pre-supply one on command line
12:56 <@zyga> spineau: we could get one from hexr or somewhere like that
12:56 <@spineau> zyga: to me this could be done dynamically, before
the test selection to allow the tester to manually specify the #ports
12:56 <@zyga> spineau: or we could ask the user before all testing is
started and cache it
12:56 <@spineau> same idea same second
12:57 <@zyga> spineau: we don't want to ask the user over and over as
this is static data
12:57 <@zyga> spineau: that's why I want to treat it differently than
resource jobs that run via dependencies
12:57 <@zyga> spineau: one more thing
12:57 <@zyga> spineau: I think we could use a "info" job/unit
12:57 <@zyga> spineau: or some other setup support
12:58 <@zyga> spineau: I see an unit that says "dear tester, this
machine has a thunderbolt port, please connect it like this (picture)"
12:58 <@zyga> spineau: and those would run before all testing as well
12:58 <@zyga> spineau: or if we go for intra-testing, those would be
just jobs we can put anywhere we want
12:58 <@zyga> spineau: those would not be a test of any kind
12:58 <@zyga> spineau: last idea
12:58 <@zyga> spineau: a support hardware unit
12:59 <@zyga> spineau: so a test that needs to do daisy chain can
indicate it needs this hardware
12:59 <@zyga> spineau: and can be skipped if the operator doesn't have
the hardware
12:59 <@zyga> spineau: this would be exclusive to stuff that's not
being tested, external hardware like a usb disk
12:59 <@zyga> spineau: and we could collect those and tell the user
before testing
12:59 <@zyga> spineau: exactly what they will need
12:59 <@zyga> spineau: adding those units is very simple
13:00 <@zyga> spineau: adding support in checkbox touch is interesting
13:00 <@zyga> spineau: what do you think?
13:00 <@spineau> zyga: I like the idea of computing hw req based on
the testplan content
13:00 <@zyga> spineau: (blank cds, monitors, cables, whatever is needed)
13:00 <@zyga> spineau: this could help a lot in making the process better
13:01 <@spineau> zyga: I'm a bit more skeptical about the manifest
unit and how to pass it to a gui for example and how to decide when we
need one without first running a cert plan of the device
13:02 <@zyga> spineau: we always need it
13:02 <@zyga> spineau: we may not use it
13:02 <@zyga> spineau: but it has to be filled in
13:03 <@zyga> spineau: it would not be passed to the gui, the system
would load one from a database (in real testing in taiwan)
13:03 <@zyga> spineau: or we'd make one once for each of our development systems
13:03 <@zyga> spineau: all we need is something that can be used as a
key (a CID perhaps)
13:03 <@spineau> zyga: but for cdts offline testing that won't scale
13:04 <@zyga> spineau: why not?
13:04 <@spineau> zyga: unless someone knows that this system MAY fail
resources detection
13:04 <@zyga> spineau: for cdts the tester will just create the
manifest first time
13:04 <@zyga> spineau: no noe
13:04 <@spineau> zyga: so someone has to run and review a first testrun
13:04 <@zyga> spineau: no
13:05 <@zyga> spineau: manifests replace resources detection for
things that we deem appropriate
13:05 <@zyga> spineau: there is no detection
13:05 <@zyga> spineau: lack of a manifest won't mean that hardware is gone
13:05 <@zyga> spineau: it will prompt the user to create the manifest
interactively
13:05 <@zyga> spineau: and cache it
13:06 <@zyga> spineau: we can create a small CLI tool for that
13:06 <@zyga> spineau: so that we won't have to touch checkbox-gui
13:06 <@zyga> spineau: but it should be a part of normal flow on
checkbox-touch and all non-frozen codebases
13:06 <@zyga> spineau: for certification in the labs we can have a
different manifest delivery mechanism (some local database from hexr)
13:07 <@spineau> zyga: one thing we could do is a tool/step/screen to
list what we automatically see and the tester can decide to add more
items (and then cache the result)
13:07 <@zyga> spineau: no, I disagree, we fundamentally need to stop
detecting flaky things
13:07 <@zyga> spineau: users will just ignore that and bad data will stay
13:07 <@zyga> spineau: and this has to be correct
13:07 <@spineau> zyga: but several resources jobs are reliable
13:07 <@zyga> spineau: so those stay as resources
13:07 <@zyga> spineau: this is for stuff like media keys
13:08 <@zyga> spineau: physical ports (do you really have that hdmi0
or is it just dead unsoldered crap on the m/b)
13:08 <@zyga> spineau: things that we can detect reliably should stay as such
13:08 <@zyga> spineau: this is for real-life observable elements of the device
13:08 <@zyga> spineau: that we cannot probe
13:09 <@zyga> spineau: leds, keyboard arrangment, connectors, form factor, etc)
13:09 <@spineau> zyga: IIRC we should also add 80211 ac as we found
one system that didn't advertise itself as compatible
13:09 <@zyga> spineau: firmware will be broken and will always lie or
not tell in general, if our testing depends on it we should not rely
on a flaky component
13:09 <@zyga> spineau: yep
13:09 <@zyga> spineau: and
13:10 <@zyga> spineau: a subset of the manifest can be used on the
certification website
13:10 <@zyga> spineau: as icons/badges etc
13:10 <@zyga> spineau: e.g. thunderbolt icon
13:10 <@zyga> spineau: usb 3.0 icon
13:10 <@zyga> spineau: whatever we want to use there
13:10 <@zyga> spineau: the point is that this is something that is unchangable
13:11 <@zyga> spineau: we need to be careful with servers and other
machines that have lots of add-on cards
13:11 <@zyga> spineau: but for phones, tables and laptops
13:11 <@zyga> spineau: this is perfect
13:11 <@zyga> spineau: screen size, amount of memory, available
connection systems (modem type)
13:11 <@zyga> spineau: all of that stuff
13:11 <@zyga> spineau: actually
13:11 <@zyga> spineau: this also nicely blends with one core addition
I wanted to do
13:11 <@zyga> spineau: static resources
13:11 <@zyga> spineau: so that we instantiate templates staitically
13:12 <@zyga> spineau: and verify that
13:12 <@spineau> zyga: let's keep this idea opened, and restart the
discussion tomorrow as I have to go
13:12 <@spineau> sorry
13:12 <@zyga> spineau: like the selftest provider that statically
tells which python libraries we have
13:12 <@zyga> spineau: ok :)
13:12 <@zyga> spineau: thanks!
13:12 <@zyga> spineau: I'll send this to the ML
13:12 <@spineau> zyga: thanks to you
13:12 <@spineau> ok
13:12 <@spineau> bye

Best regards
ZK