ubuntu-phone team mailing list archive

[Olivier Tilloy <olivier.tilloy@xxxxxxxxxxxxx>]

On Wed, Dec 11, 2013 at 6:29 PM, Paul Larson <paul.larson@xxxxxxxxxxxxx>wrote:

> On Wed, Dec 11, 2013 at 10:28 AM, Olivier Tilloy
> <olivier.tilloy@xxxxxxxxxxxxx> wrote:
> > Is it common for the webbrowser-app tests to fail, be re-run, and then
> pass?
> > Does that happen weekly, daily, more often? Has it always been the case,
> or
> > did it start happening recently?
> > When it does happen, does the second run always pass, or how many re-runs
> > are necessary to reach the green state?
> Not *super* common, but it does happen from time to time. Last week,
> we re-ran a couple of webbrowser tests.  All tests have become
> considerably more stable since the mtp fixes went in.  Here are a
> couple with failures that we reran and got  better results the second
> time last week:
>
> http://q-jenkins.ubuntu-ci:8080/job/trusty-touch-maguro-smoke-webbrowser-app-autopilot/47/
>
> http://q-jenkins.ubuntu-ci:8080/job/trusty-touch-mako-smoke-webbrowser-app-autopilot/57/
>
>
> >> For instance, here is on image 57 webbrowser with a lot of failing
> tests:
> >>
> http://ci.ubuntu.com/smokeng/trusty/touch/mako/57:20131210.1:20131203/5375/webbrowser-app-autopilot/
> .
> >> Would be interesting to know the root case, it seems something is
> screwed
> >> at some point, and then all tests are failing, can you have a look at
> that
> >> case and try to be more robust? (or pass to the CI team if it's an infra
> >> issue).
> >
> >
> > This specific run looks like an infrastructure issue to me, after a given
> > test all the subsequent tests fail when starting the app (it might be
> that
> > the app actually starts, but autopilot fails to find it). There are no
> crash
> > files in the artifacts, so it doesn’t look like an issue in the app
> itself.
> +Thomi - is there any way to tell the difference between the app not
> being started and autopilot not finding it?
> Unfortunately I don't have much to go on here either, but the maguro
> test of the same image passed fine and several builds around it
> passed, so it seems likely that mako would have passed if we reran it.
> Do the webbrowser tests depend on external websites or does it just
> connect locally on the device?
>

The tests spawn a local test server, no connection to the outside world is
made.