ubuntu-phone team mailing list archive

[Sergio Schvezov <sergio.schvezov@xxxxxxxxxxxxx>]

On Fri, May 16, 2014 at 6:42 AM, Martin Pitt <martin.pitt@xxxxxxxxxx> wrote:

> Hey Nicholas,
>
> I'd like to mention a third method of providing tests, their
> dependencies, and running them: autopkgtests. They also certainly
> don't fully match your requirements, but I believe they might be an
> appropriate candidate.
>
> Nicholas Skaggs [2014-05-13 18:56 -0400]:
> > 1.  No dependency resolution.
>
> autopkgtest does have these, you can specify test(only) dependencies
> in debian/tests/control's Depends:.
>
> > 2.  Provisioning relies on hacky scripts and/or domain specific
> > knowledge.
>
> Obviously one needs to know how to run an autopkgtest. But it's fairly
> well documented now, and as we have quite a large number of them these
> days that knowledge has become fairly well spread by now.
>
> > 3.  External Dependencies. We implicitly depend on things in the
> > image that are not really part of the sdk / ubuntu platform.
> > Autopilot in particular releases outside of the scope of the sdk and
> > is not part of it.
>
> So with "external" you mean "not installed on the phone image"? Isn't
> that just the same as 1. then?
>
> > whatever solution we decide to use must have a few key features:
> >
> >    Must allow test suites to be packaged and linked to a particular
> >    version of an application.
>
> The tests are part of the same source that builds the actual packages.
> We don't currently enforce that the binaries that the test installs
> come from the exact same source package version; but that sounds like
> a good thing to support.
>
> >  * Must ensure that test suite dependencies are always present after
> >  the test suite has been installed
>
> Check.
>
> >  *  Must allow test suite authors to easily change their test suite
> >  dependencies, without relying on a group of "gatekeeper' engineers
> >  to do this for them.
>
> That's the same as changing the source package or editing a test. TBH
> I don't think that this is a reasonable requirement to make -- it
> would mean that you have to specify the dependencies entirely separate
> from the project source and its test suite, and then have a really low
> barrier for changing it; that doesn't sound like a good idea to me.
>
> It would also mean that tests are not self-contained.
>
> >  * Must provides a unified method of installing and running tests
> >    across all phablet devices *and* the desktop.
>
> Obviously true for the desktop, but does not work for phone ATM. The
> main reason is that autopkgtest does not currently know about click
> packages; there hasn't been much desire/push for that, but I'm happy
> to look into this if we need this.
>
> And of course this would currently only work for r/w mode as you need
> to pull in deb packages for your test dependencies (like autopilot).
>
> >    Must work equally well for Canonical-developed apps and for
> >    community-developed apps. Must not rely on services available to
> >    Canonical employees only.
>
> Check.
>
> > Use Debian Packages
> >
> > All of the core apps on the phone are available as debian packages
> > (or could be). The autopilot tests have a separate binary package
> > and installing them will both ensure the app is installed and take
> > care of all the dependencies. The problem with this approach is that
> > we don't use debian packaging in general on our phablet images, and
> > by default root is locked into r/o mode.
>
> Right, this is even worse than autopkgtest, and on top of that it's
> also archive clutter and unnecessary packaging work -- there is no
> inherent need to have the tests packaged, it's enough to know how to
> get them and have a standardized format for their metadata.
>
> > Using Click Packages
> >
> > The other way we can package tests is to simply make them another
> > click package. We would need to package the testsuite, the app, and
> > all the dependencies as well as part of this process. For me this
> > includes things like autopilot, although as above, perhaps autopilot
> > and python can be considered part of the platform (thus we can
> > 'depend' on them being installed). The problem with this approach is
> > that we need to duplicate a whole lot more infrastructure, and we'd
> > be duplicating a whole lot of data (since we'd now be carrying two
> > click packages for every app).
>
> I really don't like bundling all your test dependencies in a single
> click. That enormously blows up the size of your tests, makes it
> hard to roll out updates to the test infrastructure, and also makes it
> hard to run existing tests against new versions of test dependencies
> (e. g. to check if a new autopilot version is still working with
> existing tests).
>
> I think we should just accept that we need to switch to r/w for
> installing test dependencies. For a few common things like autopilot
> we can certainly see how to run them without dpkg-installing -- like,
> unpack them in /tmp and set $PATH, $PYTHONPATH etc. But it's rather
> limiting to say "your tests can't have apache/dbusmock/samba/lcov/etc"
> just because they are only available as .deb package.
>
> If r/w is not wanted for whatever reason, then the possible test
> dependencies will be very limited. But that's independent on how you
> ship the tests (deb/click/autopkgtest).
>

If updates keep working, this r/w solution is fine. I think the solution
should target application authors who want to test on their upcoming BQs
and Meizu's as well. When you start using your Ubuntu Touch device as your
main phone/tablet, these things start rolling into your mind.