unity-api-bugs team mailing list archive

[Leo Arias <leo.arias@xxxxxxxxxxxxx>]

What must be tested on ubuntu-app-launch are all the possible ways it
can be called from the point of view of a client like autopilot. We can
either just test that u-a-l is calling upstart with the right parameters
and put tests in upstart for the actual result, or we can do a full
stack test on ual without mocks.

What must be tested on autopilot are all the launcher helpers that we
can call from a test case.

As we control these stack of projects, the toolkit should be safe with
that and doesn't require to do more integration tests. It can just
assume that if it calls AutopilotTestCase.launch*, it will get a
launched app with a testability proxy in return. When that assumption
fails, we report a bug with the info from the logs and won't be to land
anything until it's fixed. As it's a regression, it should be critical
and quick to fix.

I think that what's essential is that every time a bug escapes our
testing, the fix comes with a low level test on the same project so it
doesn't regress. If we are consistent on this, things will be always a
little easier for all the client projects.

For bugs that are not easy to reproduce, like this one, it doesn't make
a lot of sense to try lo launch an app 100 times and see if all of them
pass. What makes sense is to collect useful information in logs so if a
bug appears in a client project we can report it, reproduce it, fix it
and automate a test for it. We are not aiming to be free of bugs, we are
aiming to be free of regressions; so this case where the toolkit found a
bug in upstart shows that we have a coverage hole and can be used as a
great opportunity to close it.

I'm not sure if we have full coverage on ubuntu-app-launch with tests
from the point of view of the clients. If we miss that, then the idea of
testing in isolation all the projects we own can't work and we will have
to do manual testing and run automated tests from other projects as the
case that Tim mentioned. Maybe Ted can comment on the types of tests
that u-a-l has, and why this bug escaped the process and appeared too
many levels downstream on the toolkit.

Now in addition to that idea of having isolated tests, if all these
tests are run as autopkgtests, then every time one of the projects tries
to put a new version on the ubuntu archive, our CI machinery will run
the tests of its reverse dependencies to make sure that at the distro
level we are still consistent and don't regress. This part has pending
work on most of our projects. I think that will solve Ted's concerns of
spending a lot of time running external tests, and Tim's concern about
not being able to rely on the libraries he's using. Let me know what do
you think.

-- 
You received this bug notification because you are a member of Unity API
bugs, which is subscribed to Ubuntu Application Launcher.
https://bugs.launchpad.net/bugs/1358356

Title:
  Run ubuntu-ui-toolkit tests before landing

Status in Ubuntu Application Launcher:
  Incomplete

Bug description:
  Please run UITK autopilot tests before landing changes to launcher to
  ensure it didn't break.

  This will avoid introducing issues such as this bug:
  https://bugs.launchpad.net/ubuntu-app-launch/+bug/1357252

  To run the UITK tests, execute the following (copied from the bug
  above):

  [terminal 1]
  1. flash/update device if needed
  2. adb shell
  3. apt install ubuntu-ui-toolkit-autopilot
  4. powerd-cli display on bright # leave running, also unlock the screen so that lenses are shown
  [terminal 2]
  5. phablet-config autopilot --dbus-probe enable # wait until finishes
  6. phablet-test-run ubuntuuitoolkit

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu-app-launch/+bug/1358356/+subscriptions