ubuntu-phone team mailing list archive

[Ursula Junque <ursinha@xxxxxxxxxxx>]

On Fri, Nov 29, 2013 at 10:01 AM, Alexander Sack <asac@xxxxxxxxxxxxx> wrote:

> On Fri, Nov 29, 2013 at 12:41 PM, Oliver Grawert <ogra@xxxxxxxxxx> wrote:
> > hi,
> > On Fr, 2013-11-29 at 11:32 +0100, Alexander Sack wrote:
> >> Hi,
> >>
> >> it seems you put a few changes up for discussion in one shot.
> >>
> >> Let's keep those separate and look at them one by one:
> >>
> >> >From what I see you basically propose three main things:
> >>
> >>  1. lets increase velocity of image production so we get 2-3 images
> >> produced in devel-proposed per day
> >>  2. make cron the technology we use to schedule and kick those images
> >> 2-3 times a day
> >>  3. increase manual testing done before "releasing" images create a
> >> broader touch-release team that will include avengers and manual
> >> testers and community etc.
> >>
> >> Let me look at them one by one and then give a bullet summary of what
> >> I believe we should indeed tweak for now...
> >>
> >> On 1.
> >> ======
> >>
> >> I think 1. is and was the goal. So I think noone disagrees with the
> >> benefits of having 2-3 checkpoints a day and we should just do it.
> >> Note: it actually always was that way when I ran the landing team and
> >> during release time. I believe we still do it, but if we don't we
> >> should certainly ensure that we get back to do this.
> > on the majority of days in the past we only had one image build per day
> > simply because there were to many landings to wait for and in the end we
> > had huge change sets that burned a lot of manpower when searching where
> > a regression comes from.
> >
>
> Let's fix that process problem first.
>
> All we need to do is to be strict about following the time windows for
> cutting images regardless of whether the image has a chance to get
> promoted or not. We haven't spelled things out like this before, so I
> am pretty confident that this discussion helped getting us there.
>
>
> >>
> >> On 2.
> >> ======
> >>
> >> You are suggesting a technical solution to the problem "how and when
> >> do we cut images".
> >>
> >> I don't see why we would go for cron if we have something that is
> >> smarter - e.g. our landing process. It would be a big step back to do
> >> that. Let's be smarter :)...
> >>
> >> What we did during the final weeks of release and what we should
> >> continue to do (until we have trigger based image production) was to
> >> cut images based on a smart, individual landing plan that doesn't use
> >> a strict time approach, but rather a hybrid approach that also takes
> >> landing goals into account also
> >>
> >> For instance, every morning, landing team looks at the work to do and
> >> decides what chunks of work we would like to have in image 1,2,3...
> >> then they set themselves a hard end time to avoid that we drag on
> >> without images forever. This worked pretty well.
> >>
> >> On top we should ensure that we continue producing images also during
> >> times where landing team does not operate. That's mostly on weekend,
> >> but also might be during eur/US nights. For those times we can use
> >> cron to compensate the lack of available brains :0
> >>
> >
> > we should have a fixed cron schedule even if the landing team is around,
> > it is a huge pain if the change sets get bigger, how about we have one
> > or two fixed cron builds per day and still the opportunity to trigger a
> > third manual build at will. (the testing infrastructure is still highly
> > unstable and unreliable, tests need to be re-run on nearly every image
> > build, we have two persons doing this in two time zones and just started
> > to discuss a cron schedule on IRC that makes sure the images are built
> > at a time most convenient for them so we can have images ready during
> > their working hours with enough wiggle room for manually restarting the
> > individual tests that failed or were flaky)
>
> So you don't trust the landing team that they can make and communicate
> a predictable "time window schedule" for cutting images and follow
> that schedule? I totally do believe they can and will do it :)


> With that, I can't really see how can you still be unhappy about what
> I propose: we get the goodness of both worlds -> guaranteed frequency,
> predictability, smartness. perfect!
>

Can you show me cases where this gatekeeping prevented problems from
happening? I can point you to a few occurrences this week where we couldn't
tell where a problem is, firstly because of the amount of changes between
one image and another and secondly, because it's not deterministic that a
regression is in the chunk of changes between images. There was one image
this week that everyone was wondering why it was much worse than the
previous with the small subset of changes, all unrelated to the problems
found. There are other things we can (and need to) do to improve the
quality of a generated image, I really don't believe adding a manual gate
in the generation is helping much. But you can prove me wrong.


>
>
> >
> >
> >> On 3.
> >> ======
> >>
> >> Your proposal means very different things based on what you call
> >> "image release". So far we have used the word "promotion" to describe
> >> the act of moving a "blessed" image from a -proposed channel to a
> >> non-proposed channel. I am not sure if thats what you call "release"
> >> in your mail, but I assume so...
> > yes, i mean promoting images from -proposed to devel/trusty
> >
> >>
> >> Let's look at the channels and its purposes again:
> >>
> >>  - devel-proposed -> here all images get spit out. they are completely
> >> untested and haven't even run through automation (read: why do you
> >> want to bother big dogfooders and avengers by telling them to test
> >> this stuff)
> > because lots of regressions go out unnoticed, these images see automated
> > tests in a system that isn't very reliable yet, beyond that they get a
> > minimal smoke test (usually done by popey and me) that only covers as
> > much as we invest time ...
> > that method is not covering any regressions that show up after a while
> > only or that a manual smoketest simply didn't catch.
> >
> > we have a big community of people out there running the -proposed image
> > (I would say even more than people that actually use the devel channel),
> > we should give them a platform to be able to give us feedback and
> > participate in testing and bug triage for better regression detection.
> > locking them out by having team-only hangout meetings can't be the
> > solution to open development IMHO, lets open up to the community again
> > please.
> >
> >>  - devel -> here we put images that have gone through automation and
> >> that are ready for dogfooders to pick up
> >>  - stable -> here is where we have end users and deliver updates to
> >> end users through it.
> >>
> >> Now the consent on target frequency of those is:
> >>
> >>  - devel-proposed == 2-3 times a day (automated testing only)
> >>  - devel == 1+ times a day (dogfooders and avengers testing with goal
> >> to drive us to next stable update)
> >>  - stable == 1-6 monthlty (stable users will give even more "testing")
> >>
> >> I think that all makes sense, and doesnt' really need changing?
> > given that our automated tests cant even catch any GSM and SMS issues I
> > don't see how this all "makes sense". we have people out there using
> > these images, lets get their feedback, have them help and
> > participate ...
> >
> >>
> >> What needs better organization is the testing of dogfooders and
> >> avengers of "already blessed" devel images. Here your idea about a
> >> touch-release team makes sense. So far we had delegated that to jfunk.
> >> You could help him organize a more effective avengers effort that also
> >> includes the community, so maybe talk to him.
> >
> > how does that help at all to prevent us from promoting images
> > with regressions ?
> > having the avengers test the images is nice and all and will give us a
> > good set of high level bugs but it does not at all help with the issue
> > that we need to improve the promotion process ... automation can only
> > cover a small part here, lets involve our community in our processes
> > when we can ...
> >
> > getting bugs only from the avengers for slipped regressions also means
> > there is quite a delay sometimes so the respective package/code base has
> > evolved a lot already and when we try to nail down the issue we need to
> > do archeology. I was hoping that we could win some agility back with my
> > proposal of a public facing touch-release team, our current processes
> > are very slow and add a lot of delay everywhere while not really
> > improving the quality IMHO.
> >
> >>
> >> -----
> >>
> >> OK, let's summarize what we got so far and let's do the following
> >> tweaks for now...
> >>
> >>
> >> Summary
> >> =========
> >>
> >>  1. we start producing 2 images a day until end of year at a
> >> predicable schedule (didrocks will announce that schedule after
> >> discussing internally)
> >>
> >>  2. we don't enable cron during business days. Instead we hook image
> >> kicks up to our landing process so that we get a smart, but predicable
> >> schedule
> >>     - for instance, the times of image build will always happen around
> >> the same hours (e.g. image 1: 1200-1400, image2: 1800-2000) the same
> >> timeframe, but also will be smart about considering the landing
> >> payload so we can ensure that the critical pieces really landed etc.
> >
> > why does that matter at all ?
> >
> > if the landing is ready it enters the archive and will be automatically
> > in the next build, no matter when that build was done. the proposed
> > migration of the archive makes sure the set of packages will land
> > together (if you have packages slipping through then there is a
> > packaging bug that needs fixing)
> > I never saw (and still don't see) why image production should be tied
> > into landings at all, landings are held back in the infrastructure
> > automatically until they are complete.
> >
> >>
> >>  3. to keep the image frequency acceptable at all times, we enable
> >> cron builds during weekend and days where landing team is not
> >> operational.
> > ...
> >>
> >>  4. ogra and team to help jfunk to organize a more vibrant avengers
> >> community around testing of images after devel promotion; this team
> >> has the goal to identify issues that would block a stable promotion
> >> and will be fed back into the landing team so they can prioritize
> >> landings with the goal to clear a new stable promotion.
> >
> > you missed my most important point of involving the community by having
> > open and regular IRC meetings.
> > testing *after* promotion ... while it is nice and generates good
> > bugs ... doesn't help at all with preventing regressions from slipping
> > into the promoted images.
> >
> > note that all images we promoted since r10 had and still have
> > regressions (not to mention that the automated test pass rate is far
> > below r10 too)
> >
> > so to summarize from my side:
> >
> > 1+2+3) lets go with a semi automatic schedule then, so we even get
> > images if all ubuntu-cdimage members that can trigger them are run down
> > by simultaneous buses on different continents on the same day ;)
> > (I still disagree that the landing team is the right team to drive image
> > stuff and I also still think it puts extra load on them that should
> > better be invested into processing more landings per day. our
> > infrastructure is designed in a way that landings are held back until
> > they are installable, there is no need to bind image builds to landings.
> > we add an artificial blocker to the process were we have a reliable and
> > automatic one in place since years)
> >
> > 4) lets still have a more open regression testing process that involves
> > the community more to make sure regressions do not slip into promoted
> > images.
> >
> > ciao
> >         oli
>
> --
> Mailing list: https://launchpad.net/~ubuntu-phone
> Post to     : ubuntu-phone@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~ubuntu-phone
> More help   : https://help.launchpad.net/ListHelp
>



-- 
Ursinha (Ursula Junque)
ursinha@xxxxxxxxxxx
ursinha@xxxxxxxxxx
--
Ubuntu - I am because we are
--
Linux user #289453 - Ubuntu user #31144