ubuntu-phone team mailing list archive

[Christian Dywan <christian.dywan@xxxxxxxxxxxxx>]

Some comments in-line.

El 14/05/14 00:56, Nicholas Skaggs escribió:
>
> How we package and run acceptance tests has morphed over time. I'm
> concerned that we don't have a good method for provisioning and
> running these tests, in particular on phablet devices. This affects us
> from a CI and quality perspective, but it also affects our developers
> and those who target ubuntu as a platform as well (although autopilot
> is the primary context I'm speaking to here, this applies to any other
> tools we may wish to adopt for acceptance testing).
>
>
> The current situation is that some of the applications we ship for the
> phablet devices bundle their test suites in debian packages
> (historically these apps shipped as debian packages), while some do
> not ship their tests at all but rely on scripts (like
> phablet-click-test-setup) to pull the test suite from the source code
> repository directly. It is confusing that we have more than one
> approach for this, and it’s produced several issues:
>
>
> 1.
>
>     No dependency resolution.Under our current system, we have no way
>     to know what modules are required, nor how to get them or make
>     them available to the test. This causes tests to fail to run.
>
> 2.
>
>     Provisioning relies on hacky scripts and/or domain specific
>     knowledge.Authors have to know that tests and dependencies go in
>     ~/autopilot. Even using the developer friendly qtcreator,
>     provisioning and running our tests will fail if we've not setup
>     the device properly ourselves.
>
> 3.
>
>     External Dependencies.We implicitly depend on things in the image
>     that are not really part of the sdk / ubuntu platform. Autopilot
>     in particular releases outside of the scope of the sdk and is not
>     part of it.
>
>
> I would like to open the floor to discussing how we can solve this
> problem in a sustainable way. To my mind, whatever solution we decide
> to use must have a few key features:
>
>
>  *
>
>     Must allow test suites to be packaged and linked to a particular
>     version of an application.
>
>  *
>
>     Must ensure that test suite dependencies are always present after
>     the test suite has been installed (whether that’s achieved by
>     bundling dependencies or through installing extra packages or some
>     other means is an open question).
>
>  *
>
>     Must allow test suite authors to easily change their test suite
>     dependencies, without relying on a group of “gatekeeper’ engineers
>     to do this for them.
>
>  *
>
>     Must provides a unified method of installing and running tests
>     across all phablet devices *and* the desktop.
>
>  *
>
>     Must work equally well for Canonical-developed apps and for
>     community-developed apps. Must not rely on services available to
>     Canonical employees only.
>
>
> Considering those requirements, there are a few possibilities:
>
>
> Use Debian Packages
>
> All of the core apps on the phone are available as debian packages (or
> could be). The autopilot tests have a separate binary package and
> installing them will both ensure the app is installed and take care of
> all the dependencies. The problem with this approach is that we don't
> use debian packaging in general on our phablet images, and by default
> root is locked into r/o mode.
>
An existing problem is that using apt-get to install packages is
inherently racy. If you test anything other than the latest development
image dependencies will be wrong. So I disagree that it ensures anything
the way it works right now.
>
> Using Click Packages
>
> The other way we can package tests is to simply make them another
> click package. We would need to package the testsuite, the app, and
> all the dependencies as well as part of this process. For me this
> includes things like autopilot, although as above, perhaps autopilot
> and python can be considered part of the platform (thus we can
> 'depend' on them being installed). The problem with this approach is
> that we need to duplicate a whole lot more infrastructure, and we’d be
> duplicating a whole lot of data (since we’d now be carrying two click
> packages for every app).
>
You say "duplicating a whole lot of data" but the alternative is one or
several Debian packages containing the same data with more complexity.

What if hypothetically the test package for each app were uploaded to
the store so that it becomes trivial to install the correct test package
for a particular app? It would imply there's no dependencies to worry
about as it's click, it doesn't depend on gate keepers and it uses the
same infrastructure available to every developer today.
Autopilot could be part of a "test framework" analogous to the app
framework to resolve the dependency situation.
>
> Something Else?
>
> If we package the tests in some sane manner, provisioning and running
> them becomes very simple. Concerns have already been expressed over
> the plethora of tools and scripts that help with testing. It makes
> sense for it to be as simple as pushing the package, installing it,
> and calling autopilot directly to execute the tests. The test payload
> (or package) should be self-contained and easily installable by the
> host system. This should take care of all the dependency and setup issues.
>
>
> I'd like to open the floor to hearing your opinion on adopting a
> method for properly packaging tests and standardizing how they are
> provisioned and executed. Personally, I would like to see us continue
> to adopt and utilize click packages for this, but I would like to see
> this problem addressed as it's becoming a blocker for the platform
> moving forward. Thoughts?
>
>
> Nicholas
>
>
>
>

Attachment: signature.asc
Description: OpenPGP digital signature