← Back to team overview

ubuntu-phone team mailing list archive

Re: Acceptance testing in a sustainable manner

 

On Thu, May 15, 2014 at 6:39 PM, Nicholas Skaggs <
nicholas.skaggs@xxxxxxxxxxxxx> wrote:

> On 05/15/2014 03:03 AM, Christian Dywan wrote:
>
>> Some comments in-line.
>>
>> El 14/05/14 00:56, Nicholas Skaggs escribió:
>>
>>> Use Debian Packages
>>>
>>>
>>> All of the core apps on the phone are available as debian packages (or
>>> could be). The autopilot tests have a separate binary package and
>>> installing them will both ensure the app is installed and take care of
>>> all the dependencies. The problem with this approach is that we don't
>>> use debian packaging in general on our phablet images, and by default
>>> root is locked into r/o mode.
>>>
>>>  An existing problem is that using apt-get to install packages is
>> inherently racy. If you test anything other than the latest development
>> image dependencies will be wrong. So I disagree that it ensures anything
>> the way it works right now.
>>
> Since I'm not in favor of going this way, let's just add your point as
> another technical hurdle should we choose to go the debian packaging route.


+1 on the negative. You can also bring in unexpected dependencies.


>
>  Using Click Packages
>>>
>>> The other way we can package tests is to simply make them another
>>> click package. We would need to package the testsuite, the app, and
>>> all the dependencies as well as part of this process. For me this
>>> includes things like autopilot, although as above, perhaps autopilot
>>> and python can be considered part of the platform (thus we can
>>> 'depend' on them being installed). The problem with this approach is
>>> that we need to duplicate a whole lot more infrastructure, and we’d be
>>> duplicating a whole lot of data (since we’d now be carrying two click
>>> packages for every app).
>>>
>>>  You say "duplicating a whole lot of data" but the alternative is one or
>> several Debian packages containing the same data with more complexity.
>>
> I'm arguing in favor of moving to click, and I don't think duplication
> it's a big deal in this case (think of these packages as debug or temporary
> if it helps). That said, it is true that the click package containing tests
> for a random app will be larger than the debian package of the same app.
> For the click side, we need to pull in autopilot (or perhaps not as you've
> suggested below) and all the test dependencies -- and we have to do this
> for every click we generate. We don't get to re-use the same module for all
> our apps as we do with the debian packages. Put simply, the common code
> shared between the tests will be duplicated in every package.


There are similar discussion with golang going on; if autopilot has a bug
fix, you need to rebuild all your test packages.


>
>
>> What if hypothetically the test package for each app were uploaded to
>> the store so that it becomes trivial to install the correct test package
>> for a particular app?
>>
> Mmm, the store idea is interesting. That said, I'm not sure such 'meta'
> packages need to end up in the store. I suppose the goal would be to ensure
> they can be updated?


You can already upload a source click package; I'm entirely convinced, but
adding a test package might be good; but this solution doesn't cover the
full spectrum.


>
>  It would imply there's no dependencies to worry
>> about as it's click, it doesn't depend on gate keepers and it uses the
>> same infrastructure available to every developer today.
>>
> The dependency situation would still come into play for test writers and
> developers trying to run there tests. There is no method atm whereby we can
> depend on another click package. So I would still need to understand I have
> to install another click before running -- even if that click was
> 'test-depends-for-my-app'. I suppose this can be added to the build
> removing my concern.
>
>  Autopilot could be part of a "test framework" analogous to the app
>> framework to resolve the dependency situation.
>>
> Having a packaged version of autopilot is interesting - - I suppose the
> update model then stays the same as any other store app. The downside is
> you lose the ability to bundle tests with the autopilot runner of your
> choice.
>

I think with the test framework is analogous to the app framework in click;
it could be in itself just part of the app framework. That brings in the
added complexity of not breaking APIs when you are on a given framework
(unless it's -dev). It also means it's batteries included on the image; as
a 3rd party app developer I just need to create click; run the tests; no
extraneous provisioning.


> Could someone more intimate with click expound upon the idea of packaging
> something like autopilot as a click package?
>

Autopilot as a click package is just as useful as it copied into
/home/phablet/autopilot or wherever with the added need to figure out
dynamically where that click landed as the instalation path is determined
by the click framework.

References