ubuntu-phone team mailing list archive

[Nicholas Skaggs <nicholas.skaggs@xxxxxxxxxxxxx>]

How we package and run acceptance tests has morphed over time. I'm 
concerned that we don't have a good method for provisioning and running 
these tests, in particular on phablet devices. This affects us from a CI 
and quality perspective, but it also affects our developers and those 
who target ubuntu as a platform as well (although autopilot is the 
primary context I'm speaking to here, this applies to any other tools we 
may wish to adopt for acceptance testing).


The current situation is that some of the applications we ship for the 
phablet devices bundle their test suites in debian packages 
(historically these apps shipped as debian packages), while some do not 
ship their tests at all but rely on scripts (like 
phablet-click-test-setup) to pull the test suite from the source code 
repository directly. It is confusing that we have more than one approach 
for this, and it's produced several issues:


1.

   No dependency resolution.Under our current system, we have no way to
   know what modules are required, nor how to get them or make them
   available to the test. This causes tests to fail to run.

2.

   Provisioning relies on hacky scripts and/or domain specific
   knowledge.Authors have to know that tests and dependencies go in
   ~/autopilot. Even using the developer friendly qtcreator,
   provisioning and running our tests will fail if we've not setup the
   device properly ourselves.

3.

   External Dependencies.We implicitly depend on things in the image
   that are not really part of the sdk / ubuntu platform. Autopilot in
   particular releases outside of the scope of the sdk and is not part
   of it.


I would like to open the floor to discussing how we can solve this 
problem in a sustainable way. To my mind, whatever solution we decide to 
use must have a few key features:


 *

   Must allow test suites to be packaged and linked to a particular
   version of an application.

 *

   Must ensure that test suite dependencies are always present after
   the test suite has been installed (whether that's achieved by
   bundling dependencies or through installing extra packages or some
   other means is an open question).

 *

   Must allow test suite authors to easily change their test suite
   dependencies, without relying on a group of "gatekeeper' engineers
   to do this for them.

 *

   Must provides a unified method of installing and running tests
   across all phablet devices *and* the desktop.

 *

   Must work equally well for Canonical-developed apps and for
   community-developed apps. Must not rely on services available to
   Canonical employees only.


Considering those requirements, there are a few possibilities:


Use Debian Packages

All of the core apps on the phone are available as debian packages (or 
could be). The autopilot tests have a separate binary package and 
installing them will both ensure the app is installed and take care of 
all the dependencies. The problem with this approach is that we don't 
use debian packaging in general on our phablet images, and by default 
root is locked into r/o mode.


Using Click Packages

The other way we can package tests is to simply make them another click 
package. We would need to package the testsuite, the app, and all the 
dependencies as well as part of this process. For me this includes 
things like autopilot, although as above, perhaps autopilot and python 
can be considered part of the platform (thus we can 'depend' on them 
being installed). The problem with this approach is that we need to 
duplicate a whole lot more infrastructure, and we'd be duplicating a 
whole lot of data (since we'd now be carrying two click packages for 
every app).


Something Else?

If we package the tests in some sane manner, provisioning and running 
them becomes very simple. Concerns have already been expressed over the 
plethora of tools and scripts that help with testing. It makes sense for 
it to be as simple as pushing the package, installing it, and calling 
autopilot directly to execute the tests. The test payload (or package) 
should be self-contained and easily installable by the host system. This 
should take care of all the dependency and setup issues.


I'd like to open the floor to hearing your opinion on adopting a method 
for properly packaging tests and standardizing how they are provisioned 
and executed. Personally, I would like to see us continue to adopt and 
utilize click packages for this, but I would like to see this problem 
addressed as it's becoming a blocker for the platform moving forward. 
Thoughts?


Nicholas