ubuntu-phone team mailing list archive

[Dave Morley <davmor2@xxxxxxxxxxxxx>]

On 28/11/13 12:13, Oliver Grawert wrote:
> Hi,
> 
> As some might have noticed we recently had a few bad image releases into
> the Trusty channel that contained regressions. 
> We also tested out some changes to the build policies of the proposed
> images that I would like to establish further.
> 
> Within the last weeks we raised the amount of built images per day to a
> higher frequency (3 at most currently) ... the reason for this:
> currently finding a regression in an image when only building a single
> image per day results in a pretty big change set. Finding the offending
> package that introduced a regression can take hours of weeding through
> package changelogs and is often based on guesswork ... usually we are
> guessing pretty well, but this still burns a lot of developer time we
> should rather spend on fixing bugs and writing tests or features.
> 
> with building at least 3 images per day the change sets got a lot
> smaller, finding issues becomes a charm and offending packages are easy
> to identify ... 
> 
> My proposal here is to re-enable cron driven builds again. 
> For a start this has to be every 8h since our testing infrastructure can
> not cope with more frequent builds and over time (as our testing
> infrastructure improves) to get to a frequency of doing a build every
> 4h, this should give us a small enough set of changes to be a lot
> quicker in finding regressions. I would like to switch cron back on by
> end of this week on the above mentioned 8h schedule, it seems all
> involved teams agreed that this is a good plan, please speak up if this
> plan does not suit the way your team works (or speak up in support :) )

Yay!!!!  What times?  The likelihood is that each time zone will only
ever see 2 releases a day I'm assuming.  But it would also be good to
know if you should wait another 5 minutes for the next release before
downloading, especially if I'm on 3g.

> 
> The second part of my proposal goes a bit further than just adding a
> cron entry ...
> 
> Today the release of an image is handled by the Landing Team, which was
> traditionally only responsible for landing new upstream code inside the
> image ... this task alone already hogs most resourcesof the team.
> Releasing an image is usually pretty quickly decided as one topic of a
> daily meeting based on a quick glance over the automated tests and by
> asking for feedback from people that have done a short dogfooding
> smoketest ...
> 
> I personally think the release process deserves a lot more attention,
> way more input from other teams and a wider dogfooder audience. The
> Landing team should concentrate on the landings of new code, the time
> they need to invest in additional image testing is time they can not
> spend on testing new landings which slows all of us down by creating an
> artificial bottleneck now that all landings need to go through that
> team. They should be able to fully concentrate on this process instead.
> 
> What I like to throw in as a discussion point is to form a new
> "touch-release" team that consists of people from QA that bring in the
> buglist of collected bugs and triages and also checks errors.ubuntu.com
> regulary (we might need a "touch" filter there), one representative of
> the avengers team (who dogfood the stable images), someone from the
> ubuntu-cdimage team and a representative of the Landing team to give
> input and get feedback on the recent landings from actual endusers.
> 
> This team should be a public focused team holding daily IRC meetings the
> whole community can participate in (as opposed to privately held
> team-only hangouts), I know there are plenty of users of the
> trusty-proposed/devel-proposed images and I think we should really try
> to have them participate in the release process and enable them to give
> us feedback to:
> 
> a) regressions they run into that went unnoticed by the developers
> b) new bugs the QA/triage teams have not yet seen
> c) critical blockers we didn't even consider or see yet because they
> only happen after a day or two of constant usage.
> 
> With the higher frequency of proposed builds it should then be possible
> to pick the image with the best automated results from a former day (or
> even two days so testers had more time for long term testing) that is
> considered good by all participating parties and their individual
> requirements.
> 
> Lets put more focus onto the Image releases and take load of the Landing
> team to speed us all up and have the community more involved to actually
> increase the quality of our images and be truly regression free.
> 

I think that something based on these lines would make a certain amount
of sense.  I would hold off on a meeting everyday though if the
community are involved it is hard for everyone to turn up everyday
particularly with timezone constraints and instead schedule them for
Tuesdays and Thursday.  This would mean you roll out a stable-ish image
based on the meeting and leaves you with a nice stable release over the
weekend for people to really dogfood.

It might also be good, as we start the process of reducing the test
failure rate, that we set a level that we won't release under.  For
example if the overall image percentage drops below 90% it is simply
ignored till the next release. This cuts down drastically on the image
pool to release from.

Just some addition thoughts.


-- 
You make it, I'll break it!

I love my job :)
http://www.ubuntu.com
http://www.canonical.com

Attachment: signature.asc
Description: OpenPGP digital signature