ubuntu-phone team mailing list archive

[Oliver Grawert <ogra@xxxxxxxxxx>]

Hi,

As some might have noticed we recently had a few bad image releases into
the Trusty channel that contained regressions. 
We also tested out some changes to the build policies of the proposed
images that I would like to establish further.

Within the last weeks we raised the amount of built images per day to a
higher frequency (3 at most currently) ... the reason for this:
currently finding a regression in an image when only building a single
image per day results in a pretty big change set. Finding the offending
package that introduced a regression can take hours of weeding through
package changelogs and is often based on guesswork ... usually we are
guessing pretty well, but this still burns a lot of developer time we
should rather spend on fixing bugs and writing tests or features.

with building at least 3 images per day the change sets got a lot
smaller, finding issues becomes a charm and offending packages are easy
to identify ... 

My proposal here is to re-enable cron driven builds again. 
For a start this has to be every 8h since our testing infrastructure can
not cope with more frequent builds and over time (as our testing
infrastructure improves) to get to a frequency of doing a build every
4h, this should give us a small enough set of changes to be a lot
quicker in finding regressions. I would like to switch cron back on by
end of this week on the above mentioned 8h schedule, it seems all
involved teams agreed that this is a good plan, please speak up if this
plan does not suit the way your team works (or speak up in support :) )

The second part of my proposal goes a bit further than just adding a
cron entry ...

Today the release of an image is handled by the Landing Team, which was
traditionally only responsible for landing new upstream code inside the
image ... this task alone already hogs most resourcesof the team.
Releasing an image is usually pretty quickly decided as one topic of a
daily meeting based on a quick glance over the automated tests and by
asking for feedback from people that have done a short dogfooding
smoketest ...

I personally think the release process deserves a lot more attention,
way more input from other teams and a wider dogfooder audience. The
Landing team should concentrate on the landings of new code, the time
they need to invest in additional image testing is time they can not
spend on testing new landings which slows all of us down by creating an
artificial bottleneck now that all landings need to go through that
team. They should be able to fully concentrate on this process instead.

What I like to throw in as a discussion point is to form a new
"touch-release" team that consists of people from QA that bring in the
buglist of collected bugs and triages and also checks errors.ubuntu.com
regulary (we might need a "touch" filter there), one representative of
the avengers team (who dogfood the stable images), someone from the
ubuntu-cdimage team and a representative of the Landing team to give
input and get feedback on the recent landings from actual endusers.

This team should be a public focused team holding daily IRC meetings the
whole community can participate in (as opposed to privately held
team-only hangouts), I know there are plenty of users of the
trusty-proposed/devel-proposed images and I think we should really try
to have them participate in the release process and enable them to give
us feedback to:

a) regressions they run into that went unnoticed by the developers
b) new bugs the QA/triage teams have not yet seen
c) critical blockers we didn't even consider or see yet because they
only happen after a day or two of constant usage.

With the higher frequency of proposed builds it should then be possible
to pick the image with the best automated results from a former day (or
even two days so testers had more time for long term testing) that is
considered good by all participating parties and their individual
requirements.

Lets put more focus onto the Image releases and take load of the Landing
team to speed us all up and have the community more involved to actually
increase the quality of our images and be truly regression free.

ciao
	oli

Attachment: signature.asc
Description: This is a digitally signed message part