ubuntu-phone team mailing list archive

[Łukasz 'sil2100' Zemczak <lukasz.zemczak@xxxxxxxxxxxxx>]

Hello Thomi,

Right, it might sound a bit scary indeed. The main thing that Didier
wanted to say is that we want all the tests to be reliable. We no longer
do re-runs in case of tests that are failing (due to flakiness) and are
no longer allowed to release a component that has an unreliable test.
Tests that are flaky only add to the overall confusion.

Of course, as you say, the best way is to fix the test properly!
Sometimes though, as we experienced, it's either not that easy to do or
there are simply no resources available in a team. Integration test
reliability needs to be also relatively high-priority whenever an issue
related pops up - but with many other, seemingly more important tasks
queued, sometimes flaky tests stay around for too long. And since we
won't release a component with such anomalies, sometimes temporary
skipping the test is the only way to release a component. Since what use
is a test that cannot give proper results!

I guess those words were to enforce this high-priority to fixing
integration flakiness.

Good to hear about the TnT team - I guess we'll be poking you guys about
some of the problems we'll be encountering.

Thanks.

Best regards,

On 15.12.2013 20:51, Thomi Richards wrote:
> Hi,
> 
> On Wed, Dec 11, 2013 at 6:26 AM, Didier Roche <didrocks@xxxxxxxxxx> wrote:
> 
>> As we are really close to our target of 100% of tests reliably passing and
>> there is not anymore any known "can't launch app" blockers, it's time to
>> look at getting all flaky tests disabled or fixed. It's completely
>> reachable to get that green for Thursday I guess. If you can't fix a test
>> that is flaky, the test is useless, so please disable it: it's just
>> bringing noise to people trying to release an image.
> 
> 
> 
> This sentence scares me. Please don't disable tests that are failing
> without first:
> 
>  * Doing some investigation into *why* they're failing.
>  * If you're still stuck, talk to someone on the TnT team [1], we're here
> to help you fix this sort of problem.
>  * If, after all they you still cannot solve the problem, at least file a
> bug against your project that you need to re-add that test case at some
> point in the future.
> 
> 
> I'm sure nobody is suggesting this, but to my ear this sounds awfully close
> to "we're going to disable failing tests because we can't be bothered to
> fix them". Again, I'm not suggesting that anyone has this attitude, but
> it's important that our language (especially on a public mailing list)
> reflects our intention.
> 
> Cheers,
> 
> [1] The TnT team (Tools and Trust) are: Thomi Richards (thomi), Max
> Brustkern (nuclearbob), Corey Goldberg (cgoldberg) and Chris Lee (veebers).
> You can find us on #ubuntu-qa on irc.freenode.net.
> 
> 
> 


-- 
Łukasz 'sil2100' Zemczak
 lukasz.zemczak@xxxxxxxxxxxxx
 www.canonical.com