firmware-testing-team team mailing list archive

[Colin Ian King <colin.king@xxxxxxxxxxxxx>]

On 17/10/12 10:05, Brendan Donegan wrote:
> Hi,
>
> The following is a list of certification tests which use FWTS as their
> basis. This will be used as a foundation to start the conversation on
> how FWTS can be made suitable for certification needs. Note that I've
> focused on on the jobs where the output of fwts is not parsed - their
> are other tests called through the fwts_test wrapper, but they are not
> giving us issues.
>
> FWTS Invocations
>
> (power-management/hibernate_advanced : Tests basic S4)
> fwts s4 --s4-device-check --s4-device-check-delay=30
> --s4-sleep-delay=120 -r $CHECKBOX_DATA/hibernate-single.log
>
> (power-management/poweroff : Tests full power down)
> pm_test* poweroff --log-level=debug --log-dir=$CHECKBOX_DATA
>
> (power-management/reboot : Test restart)
> pm_test reboot --log-level=debug --log-dir=$CHECKBOX_DATA
>
> (stress/hibernate_30_cycles : S4 stress test to hibernate/resume 30 times)
> fwts s4 --s4-device-check --s4-device-check-delay=30
> --s4-sleep-delay=120 --s4-multiple=30 -r
> $CHECKBOX_DATA/hibernate_30_cycles.log
>
> (stress/suspend_30_cycles : S3 stress test to suspend/resume 30 times)
> fwts s3 --s3-device-check --s3-device-check-delay=30 --s3-sleep-delay=30
> --s3-multiple=30 -r $CHECKBOX_DATA/suspend_30_cycles.log
>
> There are also 250 cycle variants of these, which aren't used by us
>
> (suspend/suspend_advanced : Tests basic S3)
>    fwts s3 --s3-sleep-delay=30 -r $CHECKBOX_DATA/suspend_single.log
>    grep 'Critical failures: NONE' $CHECKBOX_DATA/suspend_single.log
>
> The grep is in place since this test is a dependency for other tests, so
> we need it to pass if the system did resume. The condition is a bit
> crude, I'm aware.
>
> Please ask further questions if needed.
>
> * pm_test is a wrapper script which calls FWTS :
> http://bazaar.launchpad.net/~checkbox-dev/checkbox/trunk/view/head:/scripts/pm_test
> <http://bazaar.launchpad.net/%7Echeckbox-dev/checkbox/trunk/view/head:/scripts/pm_test>
OK, So I've had a look at this and perhaps we should address this by 
making fwts only report failures based on the unique test failure ID 
that each failure message has.

For example:

FAILED [CRITICAL] KlogPciACPIOSCRequestFailedreturned: Test 1, CRITICAL 
Kernel message: [ 0.496505] pci0000:00: >ACPI _OSC request failed 
(AE_SUPPORT), returned
control mask: 0x0d

So, KlogPciACPIOSCRequestFailedreturned is the unique ID.

We could either have a profile specifying the messages you want to see 
*or* messages you don't want to see.

Bear in mind that fwts has several hundred unique IDs, so I'd like to 
have profiles either opt-in (report the given messages) or opt-out 
(don't report the given messages) based on unique IDs.

The bottom line is that it is too much work to retro-fit fwts to *not* 
make it run tests, but we can add a simple filter in/out profile that 
makes it NOT report failures keyed on the failure message unique ID.

How does that sound?

Colin


> _______________________________________________
> Mailing list: https://launchpad.net/~firmware-testing-team
> Post to     : firmware-testing-team@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~firmware-testing-team
> More help   : https://help.launchpad.net/ListHelp