← Back to team overview

ubuntu-phone team mailing list archive

Re: Acceptance testing in a sustainable manner

 

On 05/15/2014 03:03 AM, Christian Dywan wrote:
Some comments in-line.

El 14/05/14 00:56, Nicholas Skaggs escribió:
Use Debian Packages

All of the core apps on the phone are available as debian packages (or
could be). The autopilot tests have a separate binary package and
installing them will both ensure the app is installed and take care of
all the dependencies. The problem with this approach is that we don't
use debian packaging in general on our phablet images, and by default
root is locked into r/o mode.

An existing problem is that using apt-get to install packages is
inherently racy. If you test anything other than the latest development
image dependencies will be wrong. So I disagree that it ensures anything
the way it works right now.
Since I'm not in favor of going this way, let's just add your point as another technical hurdle should we choose to go the debian packaging route.
Using Click Packages

The other way we can package tests is to simply make them another
click package. We would need to package the testsuite, the app, and
all the dependencies as well as part of this process. For me this
includes things like autopilot, although as above, perhaps autopilot
and python can be considered part of the platform (thus we can
'depend' on them being installed). The problem with this approach is
that we need to duplicate a whole lot more infrastructure, and we’d be
duplicating a whole lot of data (since we’d now be carrying two click
packages for every app).

You say "duplicating a whole lot of data" but the alternative is one or
several Debian packages containing the same data with more complexity.
I'm arguing in favor of moving to click, and I don't think duplication it's a big deal in this case (think of these packages as debug or temporary if it helps). That said, it is true that the click package containing tests for a random app will be larger than the debian package of the same app. For the click side, we need to pull in autopilot (or perhaps not as you've suggested below) and all the test dependencies -- and we have to do this for every click we generate. We don't get to re-use the same module for all our apps as we do with the debian packages. Put simply, the common code shared between the tests will be duplicated in every package.

What if hypothetically the test package for each app were uploaded to
the store so that it becomes trivial to install the correct test package
for a particular app?
Mmm, the store idea is interesting. That said, I'm not sure such 'meta' packages need to end up in the store. I suppose the goal would be to ensure they can be updated?
It would imply there's no dependencies to worry
about as it's click, it doesn't depend on gate keepers and it uses the
same infrastructure available to every developer today.
The dependency situation would still come into play for test writers and developers trying to run there tests. There is no method atm whereby we can depend on another click package. So I would still need to understand I have to install another click before running -- even if that click was 'test-depends-for-my-app'. I suppose this can be added to the build removing my concern.
Autopilot could be part of a "test framework" analogous to the app
framework to resolve the dependency situation.
Having a packaged version of autopilot is interesting - - I suppose the update model then stays the same as any other store app. The downside is you lose the ability to bundle tests with the autopilot runner of your choice.

Could someone more intimate with click expound upon the idea of packaging something like autopilot as a click package?

Nicholas


Follow ups

References