On 17/10/12 10:05, Brendan Donegan wrote:
Hi,
The following is a list of certification tests which use FWTS as their
basis. This will be used as a foundation to start the conversation on
how FWTS can be made suitable for certification needs. Note that I've
focused on on the jobs where the output of fwts is not parsed - their
are other tests called through the fwts_test wrapper, but they are not
giving us issues.
FWTS Invocations
(power-management/hibernate_advanced : Tests basic S4)
fwts s4 --s4-device-check --s4-device-check-delay=30
--s4-sleep-delay=120 -r $CHECKBOX_DATA/hibernate-single.log
(power-management/poweroff : Tests full power down)
pm_test* poweroff --log-level=debug --log-dir=$CHECKBOX_DATA
(power-management/reboot : Test restart)
pm_test reboot --log-level=debug --log-dir=$CHECKBOX_DATA
(stress/hibernate_30_cycles : S4 stress test to hibernate/resume 30
times)
fwts s4 --s4-device-check --s4-device-check-delay=30
--s4-sleep-delay=120 --s4-multiple=30 -r
$CHECKBOX_DATA/hibernate_30_cycles.log
(stress/suspend_30_cycles : S3 stress test to suspend/resume 30 times)
fwts s3 --s3-device-check --s3-device-check-delay=30 --s3-sleep-delay=30
--s3-multiple=30 -r $CHECKBOX_DATA/suspend_30_cycles.log
There are also 250 cycle variants of these, which aren't used by us
(suspend/suspend_advanced : Tests basic S3)
fwts s3 --s3-sleep-delay=30 -r $CHECKBOX_DATA/suspend_single.log
grep 'Critical failures: NONE' $CHECKBOX_DATA/suspend_single.log
The grep is in place since this test is a dependency for other tests, so
we need it to pass if the system did resume. The condition is a bit
crude, I'm aware.
Please ask further questions if needed.
* pm_test is a wrapper script which calls FWTS :
http://bazaar.launchpad.net/~checkbox-dev/checkbox/trunk/view/head:/scripts/pm_test
<http://bazaar.launchpad.net/%7Echeckbox-dev/checkbox/trunk/view/head:/scripts/pm_test>
OK, So I've had a look at this and perhaps we should address this by
making fwts only report failures based on the unique test failure ID
that each failure message has.
For example:
FAILED [CRITICAL] KlogPciACPIOSCRequestFailedreturned: Test 1,
CRITICAL Kernel message: [ 0.496505] pci0000:00: >ACPI _OSC request
failed (AE_SUPPORT), returned
control mask: 0x0d
So, KlogPciACPIOSCRequestFailedreturned is the unique ID.
We could either have a profile specifying the messages you want to see
*or* messages you don't want to see.
Bear in mind that fwts has several hundred unique IDs, so I'd like to
have profiles either opt-in (report the given messages) or opt-out
(don't report the given messages) based on unique IDs.
The bottom line is that it is too much work to retro-fit fwts to *not*
make it run tests, but we can add a simple filter in/out profile that
makes it NOT report failures keyed on the failure message unique ID.
How does that sound?
