ubuntu-phone team mailing list archive

[Didier Roche <didrocks@xxxxxxxxxx>]

Le 04/03/2014 20:14, Thomi Richards a écrit :
> Hi,
>
> On Wed, Mar 5, 2014 at 6:30 AM, Didier Roche <didrocks@xxxxxxxxxx 
> <mailto:didrocks@xxxxxxxxxx>> wrote:
>
>     However, some unity8 tests reliably failed and Michal reported
>     it[1]. We reverted autopilot-qt and autopilot (part of the
>     transaction). We avoided reverting xpathselect which was part of
>     the transaction as well but only contained packaging changes.
>     Corey, can you ensure that unity8 AP tests are part of the
>     autopilot test plan? I think you will need to have a wiki page
>     like most projects with the autopilot test plan now and including
>     a big suit of tests to run (or add them to the autopilot tests
>     themselves).
>     Feel free to ask for another landing or reuse the same silo to
>     reland your changes once you have the fix and the tests.
>     This revert will be part of image #220.
>
>
> Unity8 is on the test schedule. We found failing tests in the unity8 
> package with the new autopilot, but after Leo and I looked into it 
> further, we were able to get the same tests to fail with the existing 
> autopilot as well. The consistently failing test was:
>
> unity8.application_lifecycle.tests.test_url_dispatcher.URLDispatcherTestCase.test_swipe_out_application_started_by_url_dispatcher(Native 
> Device)
>
> The test tries to start an app via the url dispatcher (as far as I can 
> make out anyway - please correct me if that's wrong), but often the 
> application will not start, which causes the tests to fail.
>
> We also found tests that fail intermittenly with both the new, and old 
> autopilot versions. These include:
>
> unity8.shell.tests.test_notifications.EphemeralNotificationsTests.test_icon_summary_body(Native 
> Device)
> unity8.shell.tests.test_notifications.EphemeralNotificationsTests.test_update_notification_same_layout(Native 
> Device)
> unity8.shell.tests.test_notifications.EphemeralNotificationsTests.test_icon_summary(Native 
> Device)
> unity8.shell.tests.test_emulators.DashAppsEmulatorTestCase.test_get_applications_with_unexisting_category(Native 
> Device)
>
>
> Given that we found these regressions in both versions of autopilot, 
> we made the assumption that the failure was not related to the new 
> autopilot packages. You've obviously come to a different conclusion. 
> Please let me know what I need to do to convince you to let us land 
> autopilot.

As per the bug and reproduced by 2 people + the CI dashboard, we got 
this failure (see the bug report) reliably on the new autopilot on 
unity8 and didn't get it with the previous version. I would suggest that 
you work with the Unity8 guys to ensure that's the case and work with 
them to get this test passing with the new autopilot.

The other failure is what is explained in extends in all landing 
threads, the Qt 5.0 crash which doesn't reproduce everytime. So we rerun 
multiple times to ensure that the crash doesn't hide any other 
regressions and you have to do that as well until we are out of this 
degraded landing mode.

Cheers,
Didier