ubuntu-phone team mailing list archive

[Martin Pitt <martin.pitt@xxxxxxxxxx>]

Hey Nicholas,

I'd like to mention a third method of providing tests, their
dependencies, and running them: autopkgtests. They also certainly
don't fully match your requirements, but I believe they might be an
appropriate candidate.

Nicholas Skaggs [2014-05-13 18:56 -0400]:
> 1.  No dependency resolution.

autopkgtest does have these, you can specify test(only) dependencies
in debian/tests/control's Depends:.

> 2.  Provisioning relies on hacky scripts and/or domain specific
> knowledge.

Obviously one needs to know how to run an autopkgtest. But it's fairly
well documented now, and as we have quite a large number of them these
days that knowledge has become fairly well spread by now.

> 3.  External Dependencies. We implicitly depend on things in the
> image that are not really part of the sdk / ubuntu platform.
> Autopilot in particular releases outside of the scope of the sdk and
> is not part of it.

So with "external" you mean "not installed on the phone image"? Isn't
that just the same as 1. then?

> whatever solution we decide to use must have a few key features:
> 
>    Must allow test suites to be packaged and linked to a particular
>    version of an application.

The tests are part of the same source that builds the actual packages.
We don't currently enforce that the binaries that the test installs
come from the exact same source package version; but that sounds like
a good thing to support.

>  * Must ensure that test suite dependencies are always present after
>  the test suite has been installed

Check.

>  *  Must allow test suite authors to easily change their test suite
>  dependencies, without relying on a group of "gatekeeper' engineers
>  to do this for them.

That's the same as changing the source package or editing a test. TBH
I don't think that this is a reasonable requirement to make -- it
would mean that you have to specify the dependencies entirely separate
from the project source and its test suite, and then have a really low
barrier for changing it; that doesn't sound like a good idea to me.

It would also mean that tests are not self-contained.

>  * Must provides a unified method of installing and running tests
>    across all phablet devices *and* the desktop.

Obviously true for the desktop, but does not work for phone ATM. The
main reason is that autopkgtest does not currently know about click
packages; there hasn't been much desire/push for that, but I'm happy
to look into this if we need this.

And of course this would currently only work for r/w mode as you need
to pull in deb packages for your test dependencies (like autopilot).

>    Must work equally well for Canonical-developed apps and for
>    community-developed apps. Must not rely on services available to
>    Canonical employees only.

Check.

> Use Debian Packages
> 
> All of the core apps on the phone are available as debian packages
> (or could be). The autopilot tests have a separate binary package
> and installing them will both ensure the app is installed and take
> care of all the dependencies. The problem with this approach is that
> we don't use debian packaging in general on our phablet images, and
> by default root is locked into r/o mode.

Right, this is even worse than autopkgtest, and on top of that it's
also archive clutter and unnecessary packaging work -- there is no
inherent need to have the tests packaged, it's enough to know how to
get them and have a standardized format for their metadata.

> Using Click Packages
> 
> The other way we can package tests is to simply make them another
> click package. We would need to package the testsuite, the app, and
> all the dependencies as well as part of this process. For me this
> includes things like autopilot, although as above, perhaps autopilot
> and python can be considered part of the platform (thus we can
> 'depend' on them being installed). The problem with this approach is
> that we need to duplicate a whole lot more infrastructure, and we'd
> be duplicating a whole lot of data (since we'd now be carrying two
> click packages for every app).

I really don't like bundling all your test dependencies in a single
click. That enormously blows up the size of your tests, makes it
hard to roll out updates to the test infrastructure, and also makes it
hard to run existing tests against new versions of test dependencies
(e. g. to check if a new autopilot version is still working with
existing tests).

I think we should just accept that we need to switch to r/w for
installing test dependencies. For a few common things like autopilot
we can certainly see how to run them without dpkg-installing -- like,
unpack them in /tmp and set $PATH, $PYTHONPATH etc. But it's rather
limiting to say "your tests can't have apache/dbusmock/samba/lcov/etc"
just because they are only available as .deb package.

If r/w is not wanted for whatever reason, then the possible test
dependencies will be very limited. But that's independent on how you
ship the tests (deb/click/autopkgtest).

Thanks,

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)

Attachment: signature.asc
Description: Digital signature