checkbox-dev team mailing list archive

[Zygmunt Krynicki <zygmunt.krynicki@xxxxxxxxxxxxx>]

I think this is not so useful because those detection mechanisms being
unreliable are the whole reason we need manifests.

For things that we presently detect we just keep doing what we do.
This is for things we cannot know for sure. Having an aid that "helps"
is something we might consider but I question if it is useful as a
time saver (this is a one-time operation) or valuable for the quality
of data (oooh, did I just press enter here?)

Thanks
ZK

On Wed, Mar 25, 2015 at 3:03 PM, Chris Gregan
<chris.gregan@xxxxxxxxxxxxx> wrote:
> I like this idea, but I echo spineau when it comes to detection. I'm
> wondering if there is a middle ground. So, for example, the detection runs,
> and then compares to a list we have of all the possible ports that are on
> systems typically. If the detection returns no HDMI, Webcam, and Bluetooth,
> we would then prompt the user to confirm our findings and if they find a
> detection problem then they can create a manifest.
>
> Chris
>
> On Wed, Mar 25, 2015 at 8:12 AM, Zygmunt Krynicki
> <zygmunt.krynicki@xxxxxxxxxxxxx> wrote:
>>
>> Hey
>>
>> I wanted keep everyone in the loop about thunderbolt and about the
>> idea to use manifests:
>>
>> 12:52 <@zyga> spineau: let's talk about manifests
>> 12:52 <@spineau> zyga: to handle job upgrade?
>> 12:53 <@zyga> spineau: in my earlier drafts I was proposing to add a
>> manifest entry unit. each item would have an ID, a name (as listed in
>> the manifest), a type (bool, int, we can add more as needed) and
>> optional unit type
>> 12:53 <@zyga> spineau: no, for thunderbolt
>> 12:53 <@zyga> spineau: so for TB, we could say that there's a unit like
>> this:
>> 12:53 <@zyga> unit: manifest entry
>> 12:53 <@zyga> id: number-of-thunderbolt-ports
>> 12:53 <@zyga> type: int
>> 12:54 <@zyga> _name: Number of thunderbolt ports
>> 12:54 <@zyga> _description: This unit describes the number of external
>> thunderbolt ports as visible from the POV of the user.
>> 12:54 <@zyga> (I would now rename the id to
>> number-of-external-thunderbolt-ports)
>> 12:54 <@zyga> that's it
>> 12:55 <@zyga> plainbox would see that, would load a manifest file and
>> expose it as a special resource like thing
>> 12:55 <@zyga> spineau: in jobs we could say that a job requires:
>> number-of-thunderbolt-ports > 0
>> 12:55 <@zyga> (now I would use underscore instead of dashes)
>> 12:56 <@zyga> spineau: and this could be a special requirement on
>> manifes entry or a normal requirement, not sure yet
>> 12:56 <@zyga> spineau: the actual manifest file could be rfc822 or json
>> 12:56 <@zyga> spineau: we could pre-supply one on command line
>> 12:56 <@zyga> spineau: we could get one from hexr or somewhere like that
>> 12:56 <@spineau> zyga: to me this could be done dynamically, before
>> the test selection to allow the tester to manually specify the #ports
>> 12:56 <@zyga> spineau: or we could ask the user before all testing is
>> started and cache it
>> 12:56 <@spineau> same idea same second
>> 12:57 <@zyga> spineau: we don't want to ask the user over and over as
>> this is static data
>> 12:57 <@zyga> spineau: that's why I want to treat it differently than
>> resource jobs that run via dependencies
>> 12:57 <@zyga> spineau: one more thing
>> 12:57 <@zyga> spineau: I think we could use a "info" job/unit
>> 12:57 <@zyga> spineau: or some other setup support
>> 12:58 <@zyga> spineau: I see an unit that says "dear tester, this
>> machine has a thunderbolt port, please connect it like this (picture)"
>> 12:58 <@zyga> spineau: and those would run before all testing as well
>> 12:58 <@zyga> spineau: or if we go for intra-testing, those would be
>> just jobs we can put anywhere we want
>> 12:58 <@zyga> spineau: those would not be a test of any kind
>> 12:58 <@zyga> spineau: last idea
>> 12:58 <@zyga> spineau: a support hardware unit
>> 12:59 <@zyga> spineau: so a test that needs to do daisy chain can
>> indicate it needs this hardware
>> 12:59 <@zyga> spineau: and can be skipped if the operator doesn't have
>> the hardware
>> 12:59 <@zyga> spineau: this would be exclusive to stuff that's not
>> being tested, external hardware like a usb disk
>> 12:59 <@zyga> spineau: and we could collect those and tell the user
>> before testing
>> 12:59 <@zyga> spineau: exactly what they will need
>> 12:59 <@zyga> spineau: adding those units is very simple
>> 13:00 <@zyga> spineau: adding support in checkbox touch is interesting
>> 13:00 <@zyga> spineau: what do you think?
>> 13:00 <@spineau> zyga: I like the idea of computing hw req based on
>> the testplan content
>> 13:00 <@zyga> spineau: (blank cds, monitors, cables, whatever is needed)
>> 13:00 <@zyga> spineau: this could help a lot in making the process better
>> 13:01 <@spineau> zyga: I'm a bit more skeptical about the manifest
>> unit and how to pass it to a gui for example and how to decide when we
>> need one without first running a cert plan of the device
>> 13:02 <@zyga> spineau: we always need it
>> 13:02 <@zyga> spineau: we may not use it
>> 13:02 <@zyga> spineau: but it has to be filled in
>> 13:03 <@zyga> spineau: it would not be passed to the gui, the system
>> would load one from a database (in real testing in taiwan)
>> 13:03 <@zyga> spineau: or we'd make one once for each of our development
>> systems
>> 13:03 <@zyga> spineau: all we need is something that can be used as a
>> key (a CID perhaps)
>> 13:03 <@spineau> zyga: but for cdts offline testing that won't scale
>> 13:04 <@zyga> spineau: why not?
>> 13:04 <@spineau> zyga: unless someone knows that this system MAY fail
>> resources detection
>> 13:04 <@zyga> spineau: for cdts the tester will just create the
>> manifest first time
>> 13:04 <@zyga> spineau: no noe
>> 13:04 <@spineau> zyga: so someone has to run and review a first testrun
>> 13:04 <@zyga> spineau: no
>> 13:05 <@zyga> spineau: manifests replace resources detection for
>> things that we deem appropriate
>> 13:05 <@zyga> spineau: there is no detection
>> 13:05 <@zyga> spineau: lack of a manifest won't mean that hardware is gone
>> 13:05 <@zyga> spineau: it will prompt the user to create the manifest
>> interactively
>> 13:05 <@zyga> spineau: and cache it
>> 13:06 <@zyga> spineau: we can create a small CLI tool for that
>> 13:06 <@zyga> spineau: so that we won't have to touch checkbox-gui
>> 13:06 <@zyga> spineau: but it should be a part of normal flow on
>> checkbox-touch and all non-frozen codebases
>> 13:06 <@zyga> spineau: for certification in the labs we can have a
>> different manifest delivery mechanism (some local database from hexr)
>> 13:07 <@spineau> zyga: one thing we could do is a tool/step/screen to
>> list what we automatically see and the tester can decide to add more
>> items (and then cache the result)
>> 13:07 <@zyga> spineau: no, I disagree, we fundamentally need to stop
>> detecting flaky things
>> 13:07 <@zyga> spineau: users will just ignore that and bad data will stay
>> 13:07 <@zyga> spineau: and this has to be correct
>> 13:07 <@spineau> zyga: but several resources jobs are reliable
>> 13:07 <@zyga> spineau: so those stay as resources
>> 13:07 <@zyga> spineau: this is for stuff like media keys
>> 13:08 <@zyga> spineau: physical ports (do you really have that hdmi0
>> or is it just dead unsoldered crap on the m/b)
>> 13:08 <@zyga> spineau: things that we can detect reliably should stay as
>> such
>> 13:08 <@zyga> spineau: this is for real-life observable elements of the
>> device
>> 13:08 <@zyga> spineau: that we cannot probe
>> 13:09 <@zyga> spineau: leds, keyboard arrangment, connectors, form factor,
>> etc)
>> 13:09 <@spineau> zyga: IIRC we should also add 80211 ac as we found
>> one system that didn't advertise itself as compatible
>> 13:09 <@zyga> spineau: firmware will be broken and will always lie or
>> not tell in general, if our testing depends on it we should not rely
>> on a flaky component
>> 13:09 <@zyga> spineau: yep
>> 13:09 <@zyga> spineau: and
>> 13:10 <@zyga> spineau: a subset of the manifest can be used on the
>> certification website
>> 13:10 <@zyga> spineau: as icons/badges etc
>> 13:10 <@zyga> spineau: e.g. thunderbolt icon
>> 13:10 <@zyga> spineau: usb 3.0 icon
>> 13:10 <@zyga> spineau: whatever we want to use there
>> 13:10 <@zyga> spineau: the point is that this is something that is
>> unchangable
>> 13:11 <@zyga> spineau: we need to be careful with servers and other
>> machines that have lots of add-on cards
>> 13:11 <@zyga> spineau: but for phones, tables and laptops
>> 13:11 <@zyga> spineau: this is perfect
>> 13:11 <@zyga> spineau: screen size, amount of memory, available
>> connection systems (modem type)
>> 13:11 <@zyga> spineau: all of that stuff
>> 13:11 <@zyga> spineau: actually
>> 13:11 <@zyga> spineau: this also nicely blends with one core addition
>> I wanted to do
>> 13:11 <@zyga> spineau: static resources
>> 13:11 <@zyga> spineau: so that we instantiate templates staitically
>> 13:12 <@zyga> spineau: and verify that
>> 13:12 <@spineau> zyga: let's keep this idea opened, and restart the
>> discussion tomorrow as I have to go
>> 13:12 <@spineau> sorry
>> 13:12 <@zyga> spineau: like the selftest provider that statically
>> tells which python libraries we have
>> 13:12 <@zyga> spineau: ok :)
>> 13:12 <@zyga> spineau: thanks!
>> 13:12 <@zyga> spineau: I'll send this to the ML
>> 13:12 <@spineau> zyga: thanks to you
>> 13:12 <@spineau> ok
>> 13:12 <@spineau> bye
>>
>> Best regards
>> ZK
>>
>> --
>> pes-tools-cert mailing list
>> pes-tools-cert@xxxxxxxxxxxxxxxxxxx
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/pes-tools-cert
>
>
>
>
> --
> Chris Gregan
> Project Manager
> Professional and Engineering Services
> Canonical USA Inc.
> cgregan@xxxxxxxxxxxxx
> cgregan[irc.freenode.net]
> W-781-761-9448
>
> ----
> 1024/8806032D
> E70F 7391 6C78 9B9E 6461 1CC7 B168 E1E7 8806 032D