[Ayatana-dev] Manual tests: common location, format, and style

[Tim Penhey <tim.penhey@xxxxxxxxxxxxx>]

Hi All,

During the design sprint Aurélien gave a lightning talk about having a 
standard place and format for manual tests in the various code bases.

For the purpose of this discussion a manual test is:
 * Executed by a person
 * A precise set of instructions
 * A defined outcome to determine success or failure
 * Not automated

Part of the reason for this is a lack of infrastructure around automated 
testing of some user interface elements.

Ideally we would like to be able to get to the stage where we could take 
most, if not all, of the manual tests and encode them in some form of 
automated test system.  However we also need to understand where we are 
starting from.

Having a central place where the format is defined allows for people to 
know where to find it.  Or more importantly, a script to process the 
directories for certain projects and make the manual testing available 
publicly in an HTML format (more on this later).

== Proposal ==

We have a directory in the root of any project called "manual-tests"

This directory can have subdirectories named anything sensible for the 
project.  For example, unity may have: dash, launcher, panel, music-lens 
et al.

In these directories there are plain text files, and here is the kicker:
 * We use a standard wiki format (that has a simple parser),
   like ReST, textile, markdown, creole
   (I would say moin, but it doesn't have an easy to use stand-alone 
parser).

The files group like tests.  Tests have:
 * A unique name in an H1 style heading
 * A paragraph explaining the test
 * A set of steps to reproduce the test
 * Expected output or result

Tests that fail will have bugs filed.

The directories can then be trivially parsed by a nifty python script 
and uploaded to a public site, probably under unity.ubuntu.com or 
similar to make the tests easily accessible.

Tim