ubuntu-phone team mailing list archive

[Olivier Tilloy <olivier.tilloy@xxxxxxxxxxxxx>]

On Wed, Dec 11, 2013 at 5:28 PM, Olivier Tilloy <
olivier.tilloy@xxxxxxxxxxxxx> wrote:

>
>
>
> On Wed, Dec 11, 2013 at 12:47 PM, Didier Roche <didrocks@xxxxxxxxxx>wrote:
>
>>  Le 11/12/2013 12:38, Olivier Tilloy a écrit :
>>
>>   * notes-app (Bill)
>>> One test failing on both mako and maguro:
>>>
>>> http://ci.ubuntu.com/smokeng/trusty/touch/maguro/55:20131209.1:20131203/5355/notes-app-autopilot/558827/
>>>
>>> http://ci.ubuntu.com/smokeng/trusty/touch/mako/55:20131209.1:20131203/5356/notes-app-autopilot/558857/
>>> 3 flaky tests on mako:
>>>
>>> http://ci.ubuntu.com/smokeng/trusty/touch/mako/56:20131210:20131203/5360/notes-app-autopilot/
>>>
>>
>>  I am able to reproduce the failure on maguro, although not reliably.
>> It’s being looked at.
>> I don’t have a nexus 4 handy to look into the failures on mako,
>> unfortunately.
>>
>>
>> Does anyone on your team have a nexus 4? As it seems to be flacky and not
>> being reliably reproducible, looking at the code can help already to see if
>> a race happened.
>>
>
> Bill has a nexus 4, he’ll be running the tests on his device.
> We agreed on a plan with Ugo to tackle those flaky tests as best as we
> can, starting with the ones with can reliably reproduce on maguro.
>
>
>>   * webbrowser-app (Bill)
>>> 2 flaky tests on maguro only:
>>>
>>> http://ci.ubuntu.com/smokeng/trusty/touch/maguro/56:20131210:20131203/5363/webbrowser-app-autopilot/
>>>
>>
>>  Looks like this one has been re-run, it’s all green now. The
>> webbrowser-app tests have been green for weeks (if not months) in a row. Do
>> we have access to the logs of the previous run (with the two failures
>> you’re mentioning)?
>>
>>
>> Argh, I did ask to not rerun the test to the CI team. I'll reash it.
>> Webbrowser-app is flaky for quite some time, but the CI team generally
>> rerun the tests, that's why you did see it green for weeks/months in a row.
>>
>
> Is it common for the webbrowser-app tests to fail, be re-run, and then
> pass? Does that happen weekly, daily, more often? Has it always been the
> case, or did it start happening recently?
> When it does happen, does the second run always pass, or how many re-runs
> are necessary to reach the green state?
>
>
>
>> For instance, here is on image 57 webbrowser with a lot of failing tests:
>> http://ci.ubuntu.com/smokeng/trusty/touch/mako/57:20131210.1:20131203/5375/webbrowser-app-autopilot/
>> .
>> Would be interesting to know the root case, it seems something is screwed
>> at some point, and then all tests are failing, can you have a look at that
>> case and try to be more robust? (or pass to the CI team if it's an infra
>> issue).
>>
>
> This specific run looks like an infrastructure issue to me, after a given
> test all the subsequent tests fail when starting the app (it might be that
> the app actually starts, but autopilot fails to find it). There are no
> crash files in the artifacts, so it doesn’t look like an issue in the app
> itself.
>

And by "infrastructure", what I really meant was anywhere else in the
stack. The problem could be in autopilot, in the D-Bus session, in other
components of the stack, or even in the actual infrastructure itself.
It seems this problem has also been seen with other apps’ tests recently,
it doesn’t seem to be specific to webbrowser-app.