ubuntu-phone team mailing list archive

[Martin Pitt <martin.pitt@xxxxxxxxxx>]

Hello all,

Didier Roche [2013-12-12  7:56 +0100]:
> * messaging-app (Bill):
> One flaky test on mako. (can be sdk related)
> http://ci.ubuntu.com/smokeng/trusty/touch/mako/55:20131209.1:20131203/5356/messaging-app-autopilot/558958/
> -> Seems some lead on a workaround implemented already in
> ubuntu-ui-toolkit (which doesn't work anymore)?

For the record, this is not a flaky test. It's a regression in
Mir/SDK/Qt/whereever. The test makes a downward swipe to close the OSK
but it doesn't close. This is perfectly reproducible manually on a
fresh install (tested on image 59).

I. e. if you write a message, you are stuck in that screen, and you
can never close the OSK, open the menu, and go back or do some other
action. You can't close the OSK no matter whether or not you hit Send
first or not. (Ideally the OSK would auto-close after sending, of
course, but that might be a bug in messaging-app itself or Qt/SDK
again).

It's really sad that we can detect regressions like this
automatically (similarly the Mir crashes that the dialer-app tests
still reliably reproduce), but we don't make use of that. It seems we
still manage to land central pieces of phone infrastructure without
running all our existing app tests and hold them back on regressions.
This then causes trouble like this, failures are getting ignored for
weeks and written off as "flaky test", and then nobody knows any more
what actually broke it. I'm not that involved in phone development and
I saw this happen at least three times in a few weeks.

I realize that there indeed are flaky tests which blur the utility of
comprehensive reverse-dependency testing, but these should be
considered being as important as bugs in the apps themselves. In that
state, as Didier says, they are more of a hindrance than a help as
they make ourselves ignore real test failures.

Thanks,

Martin

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)