ubuntu-phone team mailing list archive
-
ubuntu-phone team
-
Mailing list archive
-
Message #08641
Re: Acceptance testing in a sustainable manner
Hello all,
quick status update:
Nicholas Skaggs [2014-05-30 14:08 -0400]:
> After some good discussion in Malta, we are pursuing Martin's idea of using
> autopkg to provide a proper solution to this problem. The idea requires at
> least a few things:
I got some time to work on this in this week. This required some
groundwork of abstracting/factoring out the DEP-8 control parsing
bits, which are in trunk now. With that it's relatively easy to add
support for different source package formats, and I created a "click"
autopkgtest branch which adds basic click support:
http://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a%3Dshortlog;h%3Drefs/heads/click
> Modifications to the click manifest (or perhaps a standard separate file) to
> specify dependencies for the tests
> Modifications to autopkg to allow it to read and understand clicks (pull
> depends from them, provision them, etc)
Dependencies and also autopkgtest restrictions and soon also "test
classes" (different topic, but very shortly: map tests to a set of
environments that we want to run them in, based on the context). For
now I used the existing "x-test" manifest key, with a simple and more
extended syntax to specify metadata. Documentation:
http://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a=blob_plain;f=doc/README.click-tests;hb=refs/heads/click
autopkgtest now also has a simple "testclick" source/binary click
which is used for testing support for this:
http://anonscm.debian.org/gitweb/?p=autopkgtest/autopkgtest.git;a=tree;f=tests/testclick;hb=refs/heads/click
This supports specifying a path to an executable, or a shell command
directly.
I'd appreciate some feedback on the metadata format. Does that look
suitable?
With that, you can run autopkgtests for click in a basic fashion. I
created a schroot "utopic-click" with click and ubuntu-sdk-libs
installed, then this works:
adt-run tests/testclick/ tests/testclick_0.1_all.click --- schroot utopic-click
Current TODOs:
- Click source package currently has to be specified explicitly
(prior to the .click), there is no support yet for getting it from
the x-source field.
- There is no support yet for unpacking test deps in a temp dir and
setting LD_LIBRARY_PATH/PYTHONPATH/etc. to it, i. e. to make some
test depends work on r/o system images.
Also, a simple "autopilot": "tests/autopilot" test with inferred "what
do you mean?" semantics currently doesn't work. ATM you have to say
the full thing, like
"autopilot": {
"command" = "PYTHONPATH=tests/autopilot autopilot3 run ubuntu_calculator_app"
}
(and soon also some test class or restriction, to limit this to
testbeds which can actually run this)
I really don't want to hardcode this kind of semantics into
autopkgtest itself. If specifying the above command is deemed too
complicated, we should discuss how and where to provide this kind of
indirection (mapping "special" test names to some wrappers).
> An adb runner for adt
Jean-Baptiste is working on a more general adt-virt-ssh which accepts
an external small "setup" script that configures any kind of test bed
and returns an IP plus user name. This can then e. g. use adb to
activate ssh on a real device, start ubuntu-emulator, or call nova to
request a VM, etc. We'd maintain these small setup scripts in our CI
airline (as they are very specific to which type of runners we
provide), and can keep adt-virt-ssh itself in autopkgtest.
That's currently the missing link for fully testing this with
autopilotified clicks; I haven't yet found out how to make
ubuntu-app-launch work in a schroot (calling qmlscene5 works quite
fine, but u-a-l doesn't) to provide a more simple testing for that.
I'll bug Ted about that next week :)
Indeed with these two pieces we can then run any Debian or click
source test in any of the schroot/QEMU/LXC/ssh/null runners that
autopkgtest provides :)
Martin
--
Martin Pitt | http://www.piware.de
Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
References