ubuntu-phone team mailing list archive

[Thomi Richards <thomi.richards@xxxxxxxxxxxxx>]

Hi,

On Wed, Mar 5, 2014 at 6:30 AM, Didier Roche <didrocks@xxxxxxxxxx> wrote:

> However, some unity8 tests reliably failed and Michal reported it[1]. We
> reverted autopilot-qt and autopilot (part of the transaction). We avoided
> reverting xpathselect which was part of the transaction as well but only
> contained packaging changes.
> Corey, can you ensure that unity8 AP tests are part of the autopilot test
> plan? I think you will need to have a wiki page like most projects with the
> autopilot test plan now and including a big suit of tests to run (or add
> them to the autopilot tests themselves).
> Feel free to ask for another landing or reuse the same silo to reland your
> changes once you have the fix and the tests.
> This revert will be part of image #220.
>

Unity8 is on the test schedule. We found failing tests in the unity8
package with the new autopilot, but after Leo and I looked into it further,
we were able to get the same tests to fail with the existing autopilot as
well. The consistently failing test was:

unity8.application_lifecycle.tests.test_url_dispatcher.URLDispatcherTestCase.test_swipe_out_application_started_by_url_dispatcher(Native
Device)

The test tries to start an app via the url dispatcher (as far as I can make
out anyway - please correct me if that's wrong), but often the application
will not start, which causes the tests to fail.

We also found tests that fail intermittenly with both the new, and old
autopilot versions. These include:

unity8.shell.tests.test_notifications.EphemeralNotificationsTests.test_icon_summary_body(Native
Device)
unity8.shell.tests.test_notifications.EphemeralNotificationsTests.test_update_notification_same_layout(Native
Device)
unity8.shell.tests.test_notifications.EphemeralNotificationsTests.test_icon_summary(Native
Device)
unity8.shell.tests.test_emulators.DashAppsEmulatorTestCase.test_get_applications_with_unexisting_category(Native
Device)


Given that we found these regressions in both versions of autopilot, we
made the assumption that the failure was not related to the new autopilot
packages. You've obviously come to a different conclusion. Please let me
know what I need to do to convince you to let us land autopilot.

Cheers,

-- 
Thomi Richards
thomi.richards@xxxxxxxxxxxxx