widelands-dev team mailing list archive

[Hans Joachim Desserud <launchpad@xxxxxxxxxxxx>]

Hans Joachim Desserud has proposed merging lp:~hjd/widelands/tests-poc into lp:~widelands-dev/widelands/debian.

Requested reviews:
  Widelands Developers (widelands-dev)

For more details, see:
https://code.launchpad.net/~hjd/widelands/tests-poc/+merge/250533

I have some good news and bad news:
I have added support for running the integration/regression tests with the Debian package using autopkgtest. Unfortunately the Launchpad PPA builders doesn't support this [1]. :(

Autopkgtest, also known as DEP8 (see [2] for more info), is a way to test Debian packages in order to verify they run and behave as expected. The tests are run on binary packages which means you don't have to rebuild the package each time you want to run them. This makes it easy to run automated package tests more frequently, for instance Ubuntu runs them whenever they upload a new version or when one of its dependencies change [3].

I had read a bit about this already and knew that Ubuntu/Debian was working on adding test cases to some of their core packages to ensure quality and catch regressions easier. I hadn't really looked into it, but it turned out adding autopkgtests was a lot easier than I thought. All it really needed was a control file with the dependencies and one or more tests which can then be run. (Please ignore the early commits as they were mostly horrible hacks to see if I could get *something* to run on the PPA builders).

In order to verify this, I ran it on a local VM:
$ adt-run -B --built-tree=. --- null
I checked out the widelands source code and merged this branch into it to be able to build the Debian packages. It should be possible to run the tests when building the packages, but I wasn't able to trigger it. Since the main goal was getting the tests to run and compilation took a long time, I switched to using pre-built packages instead. I added my test-PPA to get the latest packages from trunk. The "null" at the end indicates I don't want to run the tests on my system instead of through QEMU, and LXC container or similar. The command above might not be the most optimal way of running things, but it worked as a proof of concept.

As mentioned above, the Launchpad PPA builders unfortunately won't run the tests. However, they will be run for offical Debian and Ubuntu packages so when build19 is packaged, they will be run to verify Widelands works on those systems. One issue remains on this front though, which I've mentioned before, is the need for a timeout mechanism. For now I've only enable one of the test suites, because when I tried to run all tests it would freeze or get stuck on one of them thus never exiting. Ideally this should run all tests, but it would need some way to mark these tests so it doesn't end up as a process which has to be killed manully.

Please let me know if something is unclear, or if you want more details, I wasn't sure how much to include. 

[1] https://answers.launchpad.net/launchpad/+question/246354
[2] http://dep.debian.net/deps/dep8/
[3] http://packaging.ubuntu.com/html/auto-pkg-test.html#ubuntu-infrastructure
-- 
Your team Widelands Developers is requested to review the proposed merge of lp:~hjd/widelands/tests-poc into lp:~widelands-dev/widelands/debian.

=== modified file 'debian/control'
--- debian/control	2014-12-20 17:52:44 +0000
+++ debian/control	2015-02-21 15:43:19 +0000
@@ -22,6 +22,7 @@
   libglew1.6-dev | libglew-dev,
   libpng-dev, 
   zlib1g-dev
+XS-Testsuite: autopkgtest
 Vcs-Git: git://git.debian.org/git/pkg-games/widelands.git
 Vcs-Browser: http://anonscm.debian.org/gitweb/?p=pkg-games/widelands.git
 #Vcs-Svn: svn://svn.debian.org/svn/collab-maint/deb-maint/widelands/ <-- Old location of packaging

=== added directory 'debian/tests'
=== added file 'debian/tests/control'
--- debian/tests/control	1970-01-01 00:00:00 +0000
+++ debian/tests/control	2015-02-21 15:43:19 +0000
@@ -0,0 +1,2 @@
+Tests: testsuite
+Depends: @, xvfb

=== added file 'debian/tests/testsuite'
--- debian/tests/testsuite	1970-01-01 00:00:00 +0000
+++ debian/tests/testsuite	2015-02-21 15:43:19 +0000
@@ -0,0 +1,6 @@
+#!/bin/sh
+# Right now it only runs the lua_testsuite.
+# It can (and should) be expanded to run all tests, but when I tried
+# I ran in to situations where some of the tests would never finish
+xvfb-run -a --server-args="-screen 0 1024x768x24" \
+./regression_test.py -b widelands -r lua_testsuite