ubuntu-phone team mailing list archive

[Thomi Richards <thomi.richards@xxxxxxxxxxxxx>]

Hi,

On Fri, Mar 21, 2014 at 8:32 AM, Nicholas Skaggs <
nicholas.skaggs@xxxxxxxxxxxxx> wrote:

>
> As an example, the calendar blockage on the latest image has a few tests
> that started failing after 5.2 landed. You've seen the landing emails with
> this snippet;
>
> * flaky test failure on calendar-app (Alan/Jean-Baptiste):
> https://bugs.launchpad.net/ubuntu-calendar-app/+bug/1293489
>
> To fix you pull calendar trunk and investigate the tests. But wait, you
> find out there's a regression caused by qtorganizer that causes calendar
> trunk to be broken. Now before you can fix the image blocker, you need to
> fix the calendar app trunk. In the meantime nothing can land in calendar
> trunk, and no fix can land in the image. Repeat for each regression you
> find while stabilizing trunk and you can understand the source of the
> delays.


I agree - to summarise balloon's point:

One cannot assume that an acceptance test failure in application X
represents a regression in application X. Often it's actually in
low-level-component Y.

This is a symptom of having large test coverage gaps for lower level
components. It's one of the reasons why, at the recent UDS, we pushed for
decoupling the trunk branch from the release branch. At least then, under
that proposal, development teams can continue to develop their projects,
and landing managers are still responsible for releasing components without
regression to the image.

It doesn't solve the problem of "I can't release my app until the existing
failing tests are fixed", but it *does* solve the problem of "My
development team cannot (easily) write any more code while there are
failing tests in the image". The latter problem sounds like a good thing,
if you make the assumption that failing tests in app X represents a failure
in app X. As we've discovered many times in the last cycle, that's actually
reasonably rare (at least, it's rare with our test coverage being the way
it is. I'm sure this is improving already).


Cheers,
-- 
Thomi Richards
thomi.richards@xxxxxxxxxxxxx