firmware-testing-team team mailing list archive
-
firmware-testing-team team
-
Mailing list archive
-
Message #00001
Firmware Test Suite 0.12.3 available in PPA
Hi there,
I've now got the Firmware Test Suite (aka fwts) now in a PPA, so it's
available to download and play with. While packing this up I took the
liberty to change the tool's name to "fwts".
To install use:
sudo add-apt-repository ppa:firmware-testing-team/ppa-firmware-test-suite
sudo apt-get update
sudo apt-get install fwts
You also need the following tools installed to run all the tests:
sudo apt-get install iasl acpidump pciutils dmidecode
Attached is a README file on how to drive the tool, but the basics are:
to get help:
fwts --help
to run a specific test, e.g. do a DSTD and SSDT syntax check:
sudo fwts syntaxcheck
..and results get put into results.log
You can run a bunch of tests too:
sudo fwts virt mtrr klog fan battery
To see all the tests:
sudo fwts --show-tests
To run all the tests:
sudo fwts
To run all the tests except for the S3 and S4 tests:
sudo fwts --no-s3 --no-s4
To run all the tests and see what's happening:
sudo fwts --show-progress
You can also redirect the output:
sudo fwts --results-output=mylogfile
..if you specify stderr or stdout output can be directed to stderr or stderr if you so desire
I am interested if you can give it a spin and send my your results.log files to see if it
gives sensible results on a bunch of different hardware.
Whilst I've given this a fair amount of testing on various machines, I expect there are
going to be a lot of bugs in this early release, so I appreciate testing and feedback!
Thanks
Colin
README for the fireware test suite
==================================
Well, you got this far, which is a good thing.
Quick Start Guide
=================
1) Dependencies: iasl, acpidump pciutils dmidecode
Please first install iasl and acpidump:
sudo apt-get install iasl acpidump pciutils dmidecode
2) To build and install (only if building from source)
./configure
make
sudo make install
I hope to get this into a PPA once I've got the code into a more
stable form.
To get help:
fwts --help
To run all the tests (including S3 and S4 suspend/resume hibernate/resume tests) use:
fwts
..this will dump the results into results.log
Advanced Options Guide
======================
Running Specific Tests
----------------------
Run specific tests, e.g. check syntax of DSDT:
sudo fwts syntaxcheck
You can run multiple tests too:
sudo fwts syntaxcheck fan batttery
To see all the available tests:
sudo fwts --show-tests
Changing the output
-------------------
To dump the results to another file:
sudo fwts --results-output=myresults.log
or to stderr or stdout and redirect:
sudo fwts --results-output=stderr
sudo fwts --results-output=stdout > mylog.log
The log has various columns:
0001 17/05/10 15:00:48 wakealarm Test ACPI Wakealarm.
0002 17/05/10 15:00:48 wakealarm ------------------------------------------------------------------------------------------
0003 17/05/10 15:00:48 wakealarm Check existance of /sys/class/rtc/rtc0/wakealarm.
0004 17/05/10 15:00:48 wakealarm PASSED: test 1, Check existance of /sys/class/rtc/rtc0/wakealarm.
0005 17/05/10 15:00:48 wakealarm Trigger RTC wakealarm.
0006 17/05/10 15:00:48 wakealarm Trigger wakealarm for 1 seconds in the future.
0007 17/05/10 15:00:48 wakealarm PASSED: test 2, Trigger RTC wakealarm.
0008 17/05/10 15:00:48 wakealarm Check if wakealarm is fired.
0009 17/05/10 15:00:48 wakealarm Trigger wakealarm for 2 seconds in the future.
0010 17/05/10 15:00:51 wakealarm PASSED: test 3, Check if wakealarm is fired.
0011 17/05/10 15:00:51 wakealarm Multiple wakealarm firing tests.
0012 17/05/10 15:00:51 wakealarm Trigger wakealarm for 1 seconds in the future.
0013 17/05/10 15:00:53 wakealarm Trigger wakealarm for 2 seconds in the future.
0014 17/05/10 15:00:56 wakealarm Trigger wakealarm for 3 seconds in the future.
0015 17/05/10 15:01:00 wakealarm Trigger wakealarm for 4 seconds in the future.
0016 17/05/10 15:01:05 wakealarm PASSED: test 4, Multiple wakealarm firing tests.
0017 17/05/10 15:01:05 wakealarm ==========================================================================================
0018 17/05/10 15:01:05 wakealarm 4 passed, 0 failed, 0 warnings, 0 aborted.
0019 17/05/10 15:01:05 wakealarm ==========================================================================================
^ ^ ^ ^
| | | |
| | | +--- %owner (name of test that ran)
| | |
| | +--- %time
| |
| +--- %date
|
+--- %line (line number)
Also, there is an optional %field that indicates the type of message being logged:
sudo fwts wakealarm --log-format="%line %data %time %field %owner"
generates results.log containing:
0001 data 15:05:13 HED wakealarm Test ACPI Wakealarm.
0002 data 15:05:13 SEP wakealarm -------------------------------------------------------------------------------------------
0003 data 15:05:13 INF wakealarm Check existance of /sys/class/rtc/rtc0/wakealarm.
0004 data 15:05:13 RES wakealarm PASSED: test 1, Check existance of /sys/class/rtc/rtc0/wakealarm.
0005 data 15:05:13 INF wakealarm Trigger RTC wakealarm.
0006 data 15:05:13 INF wakealarm Trigger wakealarm for 1 seconds in the future.
0007 data 15:05:13 RES wakealarm PASSED: test 2, Trigger RTC wakealarm.
0008 data 15:05:13 INF wakealarm Check if wakealarm is fired.
0009 data 15:05:13 INF wakealarm Trigger wakealarm for 2 seconds in the future.
0010 data 15:05:16 RES wakealarm PASSED: test 3, Check if wakealarm is fired.
0011 data 15:05:16 INF wakealarm Multiple wakealarm firing tests.
0012 data 15:05:16 INF wakealarm Trigger wakealarm for 1 seconds in the future.
0013 data 15:05:18 INF wakealarm Trigger wakealarm for 2 seconds in the future.
0014 data 15:05:21 INF wakealarm Trigger wakealarm for 3 seconds in the future.
0015 data 15:05:25 INF wakealarm Trigger wakealarm for 4 seconds in the future.
0016 data 15:05:30 RES wakealarm PASSED: test 4, Multiple wakealarm firing tests.
0017 data 15:05:30 SEP wakealarm ===========================================================================================
0018 data 15:05:30 SUM wakealarm 4 passed, 0 failed, 0 warnings, 0 aborted.
0019 data 15:05:30 SEP wakealarm ===========================================================================================
^
|
+------------------+
|
HED = heading
INF = information fields
SEP = separator (just pretty printing)
RES = result (PASSED or FAILED)
SUM = summary
ERR = error (something fatal happened!)
WRN = warning (not a critical failure, but may need looking at)
DBG = debug output
NLN = new line (empty field)
use fwts --log-fields to show all available fields
One can filter on these fields using --log-filter, e.g.
sudo fwts --log-filter=RES,SUM
.. just collect results and summary of tests
sudo fwts --log-filter=ALL,~SEP
.. all fields except the separators
..it's quite flexible!
One can also re-arrange or modify the log format using --log-format, e.g.
sudo fwts --log-format="%line %field %time: " mtrr
this produces:
0001 HED 15:08:38: MTRR validation.
0002 SEP 15:08:38: --------------------------------------------------------------------------------------------------------------
0003 INF 15:08:38: MTRR overview
0004 INF 15:08:38: -------------
0005 INF 15:08:38: 0x00000000 - 0x80000000 write-back
0006 INF 15:08:38: 0x80000000 - 0xc0000000 write-back
0007 INF 15:08:38: 0xbf700000 - 0xbf800000 uncached
0008 INF 15:08:38: 0xbf800000 - 0xc0000000 uncached
0009 INF 15:08:38: 0xd0000000 - 0xe0000000 write-combining
0010 INF 15:08:38:
0011 INF 15:08:38: This test validates the MTRR IOMEM setup.
0012 RES 15:08:38: FAILED [MEDIUM]: test 1, Memory range 0xd0000000 to 0xdfffffff (0000:00:02.0) has incorrect attribute
0013 RES 15:08:38: write-combining
0014 INF 15:08:39: This test validates the MTRR setup across all processors.
0015 RES 15:08:39: PASSED: test 2, All processors have the MTRR setup
0016 SEP 15:08:39: ==============================================================================================================
0017 SUM 15:08:39: 1 passed, 1 failed, 0 warnings, 0 aborted.
0018 SEP 15:08:39: ==============================================================================================================
Available fields are:
%line - log line number
%time - current time
%date - current date
%owner - name of the test program (the owner of the the current test)
%field - the 3 letter record identifier field (e.g. SEP, RES, etc..)
Test Progress Feedback
======================
Use the --show-progress to dump out the test progress to stderr. Output is as follows:
mtrr : Test 1 of 2 started
mtrr : Test 1 of 2 completed (0 passed, 1 failed, 0 aborted)
mtrr : Test 2 of 2 started
mtrr : Test 2 of 2 completed (1 passed, 0 failed, 0 aborted)
^
test name ^ progress info
It's a bit crude at the moment!
PASSED or FAILED?
=================
The --stdout-summary flag outputs to stdout PASSED or FAILED if test(s) passed or failed. It's essentially
for one-shot test modes where you want to run one test and see if it passed or failed without having to
look at the results log, e.g.
sudo fwts mtrr --stdout-summary
One can override the text using the following environment variables:
BIOS_TEST_TOOLKIT_PASSED_TEXT
BIOS_TEST_TOOLKIT_FAILED_TEXT
e.g.
sudo BIOS_TEST_TOOLKIT_PASSED_TEXT="yay" fwts mtrr --stdout-summary
yay
Finally, the exit code returns 1 for test failure and 0 for test success, so the
test suite can be run from a shell script and one can detect if the test worked or not, e.g.
#!/bin/bash
LOG=stdout
sudo fwts dmi_decode --results-output=$LOG
failed=$?
Misc
====
Debug mode.. shows what the controlling test suite framework is doing. It's for
the developer only!
fwts --fwts_debug
