ubuntu-phone team mailing list archive

[Alexander Sack <asac@xxxxxxxxxxxxx>]

On Fri, Nov 29, 2013 at 1:25 PM, Oliver Grawert <ogra@xxxxxxxxxx> wrote:
> hi,
> On Fr, 2013-11-29 at 13:01 +0100, Alexander Sack wrote:
>> On Fri, Nov 29, 2013 at 12:41 PM, Oliver Grawert <ogra@xxxxxxxxxx> wrote:
>> > hi,
>> > On Fr, 2013-11-29 at 11:32 +0100, Alexander Sack wrote:
>> >> Hi,
>> >>
>> >> it seems you put a few changes up for discussion in one shot.
>> >>
>> >> Let's keep those separate and look at them one by one:
>> >>
>> >> >From what I see you basically propose three main things:
>> >>
>> >>  1. lets increase velocity of image production so we get 2-3 images
>> >> produced in devel-proposed per day
>> >>  2. make cron the technology we use to schedule and kick those images
>> >> 2-3 times a day
>> >>  3. increase manual testing done before "releasing" images create a
>> >> broader touch-release team that will include avengers and manual
>> >> testers and community etc.
>> >>
>> >> Let me look at them one by one and then give a bullet summary of what
>> >> I believe we should indeed tweak for now...
>> >>
>> >> On 1.
>> >> ======
>> >>
>> >> I think 1. is and was the goal. So I think noone disagrees with the
>> >> benefits of having 2-3 checkpoints a day and we should just do it.
>> >> Note: it actually always was that way when I ran the landing team and
>> >> during release time. I believe we still do it, but if we don't we
>> >> should certainly ensure that we get back to do this.
>> > on the majority of days in the past we only had one image build per day
>> > simply because there were to many landings to wait for and in the end we
>> > had huge change sets that burned a lot of manpower when searching where
>> > a regression comes from.
>> >
>>
>> Let's fix that process problem first.
>>
>> All we need to do is to be strict about following the time windows for
>> cutting images regardless of whether the image has a chance to get
>> promoted or not. We haven't spelled things out like this before, so I
>> am pretty confident that this discussion helped getting us there.
> why would it matter at all if an image gets promoted ... in my ideal
> world we would have builds triggered every time a change set enters from
> proposed or at least every 2h ... only a minimal amount of these images
> would be promoted at all, but we had a lot to pick the best one from ;)

The reason for that is in a CI world we feed that image back so new
merge proposals get tested on top of the most recent known good state.
And this need to happen frequently, so folks don't test on outdated
stuff etc.

And yes, we currently don't have enough automation to be sure it's
dogfood quality. But fixing that should be done through investing in
more automation rather than changing the approach to put high latency
manual and avengers activities into the middle of our rapid CI loop.


>
>>
>>
>> >>
>> >> On 2.
>> >> ======
>> >>
>> >> You are suggesting a technical solution to the problem "how and when
>> >> do we cut images".
>> >>
>> >> I don't see why we would go for cron if we have something that is
>> >> smarter - e.g. our landing process. It would be a big step back to do
>> >> that. Let's be smarter :)...
>> >>
>> >> What we did during the final weeks of release and what we should
>> >> continue to do (until we have trigger based image production) was to
>> >> cut images based on a smart, individual landing plan that doesn't use
>> >> a strict time approach, but rather a hybrid approach that also takes
>> >> landing goals into account also
>> >>
>> >> For instance, every morning, landing team looks at the work to do and
>> >> decides what chunks of work we would like to have in image 1,2,3...
>> >> then they set themselves a hard end time to avoid that we drag on
>> >> without images forever. This worked pretty well.
>> >>
>> >> On top we should ensure that we continue producing images also during
>> >> times where landing team does not operate. That's mostly on weekend,
>> >> but also might be during eur/US nights. For those times we can use
>> >> cron to compensate the lack of available brains :0
>> >>
>> >
>> > we should have a fixed cron schedule even if the landing team is around,
>> > it is a huge pain if the change sets get bigger, how about we have one
>> > or two fixed cron builds per day and still the opportunity to trigger a
>> > third manual build at will. (the testing infrastructure is still highly
>> > unstable and unreliable, tests need to be re-run on nearly every image
>> > build, we have two persons doing this in two time zones and just started
>> > to discuss a cron schedule on IRC that makes sure the images are built
>> > at a time most convenient for them so we can have images ready during
>> > their working hours with enough wiggle room for manually restarting the
>> > individual tests that failed or were flaky)
>>
>> So you don't trust the landing team that they can make and communicate
>> a predictable "time window schedule" for cutting images and follow
>> that schedule? I totally do believe they can and will do it :)
>>
>> With that, I can't really see how can you still be unhappy about what
>> I propose: we get the goodness of both worlds -> guaranteed frequency,
>> predictability, smartness. perfect!
>
> no, I trust the landing team, they are all smart people that I love to
> work with, nothing in my proposal was meant to blame anyone for
> anything ...
>
> what I dislike is that we waste manpower of our biggest bottleneck team
> where we already have reliable and well tested automation in place since
> years. which makes this artificial blocking completely unnecessary, the
> only thing we currently need to block on is the slow and unreliable
> automated testing infrastructure which has not changed much in
> reliability within the last 6 months since we started using it (there is
> still a lot of tests that have to be run manually a second time because
> their results are unreliable).
>
> our policy is to not promote regressions and our process does not live
> up to that. since we focus everything on the landing (where we wouldn't
> need to regarding image build) and not much at all into the step where
> it really matters (image promotion)
>
> ciao
>         oli