ubuntu-phone team mailing list archive

[Didier Roche <didrocks@xxxxxxxxxx>]

Le 05/03/2014 08:10, Didier Roche a écrit :
> Le 04/03/2014 20:14, Thomi Richards a écrit :
> > Hi,
> >
> > On Wed, Mar 5, 2014 at 6:30 AM, Didier Roche <didrocks@xxxxxxxxxx 
> > <mailto:didrocks@xxxxxxxxxx>> wrote:
> >
> >     However, some unity8 tests reliably failed and Michal reported
> >     it[1]. We reverted autopilot-qt and autopilot (part of the
> >     transaction). We avoided reverting xpathselect which was part of
> >     the transaction as well but only contained packaging changes.
> >     Corey, can you ensure that unity8 AP tests are part of the
> >     autopilot test plan? I think you will need to have a wiki page
> >     like most projects with the autopilot test plan now and including
> >     a big suit of tests to run (or add them to the autopilot tests
> >     themselves).
> >     Feel free to ask for another landing or reuse the same silo to
> >     reland your changes once you have the fix and the tests.
> >     This revert will be part of image #220.
> >
> >
> > Unity8 is on the test schedule. We found failing tests in the unity8 
> > package with the new autopilot, but after Leo and I looked into it 
> > further, we were able to get the same tests to fail with the existing 
> > autopilot as well. The consistently failing test was:
> >
> > unity8.application_lifecycle.tests.test_url_dispatcher.URLDispatcherTestCase.test_swipe_out_application_started_by_url_dispatcher(Native 
> > Device)
> >
> > The test tries to start an app via the url dispatcher (as far as I 
> > can make out anyway - please correct me if that's wrong), but often 
> > the application will not start, which causes the tests to fail.
> >
> > We also found tests that fail intermittenly with both the new, and 
> > old autopilot versions. These include:
> >
> > unity8.shell.tests.test_notifications.EphemeralNotificationsTests.test_icon_summary_body(Native 
> > Device)
> > unity8.shell.tests.test_notifications.EphemeralNotificationsTests.test_update_notification_same_layout(Native 
> > Device)
> > unity8.shell.tests.test_notifications.EphemeralNotificationsTests.test_icon_summary(Native 
> > Device)
> > unity8.shell.tests.test_emulators.DashAppsEmulatorTestCase.test_get_applications_with_unexisting_category(Native 
> > Device)
> >
> >
> > Given that we found these regressions in both versions of autopilot, 
> > we made the assumption that the failure was not related to the new 
> > autopilot packages. You've obviously come to a different conclusion. 
> > Please let me know what I need to do to convince you to let us land 
> > autopilot.
>
> As per the bug and reproduced by 2 people + the CI dashboard, we got 
> this failure (see the bug report) reliably on the new autopilot on 
> unity8 and didn't get it with the previous version. I would suggest 
> that you work with the Unity8 guys to ensure that's the case and work 
> with them to get this test passing with the new autopilot.
>
> The other failure is what is explained in extends in all landing 
> threads, the Qt 5.0 crash which doesn't reproduce everytime. So we 
> rerun multiple times to ensure that the crash doesn't hide any other 
> regressions and you have to do that as well until we are out of this 
> degraded landing mode.

Reintroducing the bug reference which was dropped by Thomi's post: 
https://bugs.launchpad.net/autopilot-qt/+bug/1287727. You wil note the 
new failure was: 
unity8.shell.tests.test_notifications.InteractiveNotificationBase.test_sd_incoming_call

(just to be clear that this isn't 
unity8.application_lifecycle.tests.test_url_dispatcher.URLDispatcherTestCase.test_swipe_out_application_started_by_url_dispatcher(Native 
Device) which indeed reliably fails even without a Qt 5.0 crash)

Didier