launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

So, a few days ago I put in a MP I was reviewing "RC=lifeless, if I am
able to hand out RC's" (Release-criticals).

That was well received, so I thought 'brilliant, I can help' - but -
sadly - I hadn't read
https://dev.launchpad.net/PolicyAndProcess/ReleaseManagerRotation,
which appears to be the current definition of who can approve things
RC.

So today, I was looking at a registry oops - a high frequency one -
and put together a branch, which got discussed with Curtis before
coding, reviewed by him afterwards, and approved.

I thought to myself 'well, this is all good to go, pending an RC tick,
and its trivial (it is) so low-risk, high return and easy to QA so it
won't delay the release'. Of course, thats famous last words, but so
far its holding up. I think marked it release-critical *myself* given
that Curtis and I agreed, and sent it off.

This was a problem in at least one dimension :)

Firstly, the *actual* process says that only the RM can do this; so my
tentative it-should-be-ok RC last week/weekend/whenever was actually
in error. Sorry!

I think we should review that, or explain *why* we have a bottleneck
there - we're all sensible people, and assessing how badly something
is going to affect the QA process should be rather straight forward.
(And that is what the implication of RC is - we have to QA another
patch, so there is a stall, potentially right before the release, to
do that [and if it fails QA we're left scrambling to roll-it-back as
well], and of course all this is driven by the need to coordinate
down-time with our users.

There is a newly minted process (but no docs I can find while looking
quickly - its nearly 10pm, and I wanted to get the issue up for
discussion before details fade overnight) which says (for DB patches,
but there is some differences of memory about its intended scope) that
TL's (including the TA and francis) can mutually approve things, and
that team members should go via their team lead, and failing that me,
gary or francis.

Under that process, what I did was OK as Curtis is a TL, and he agreed
that it was release critical. Under our /actual written/ process its
the wrong thing to do - and in fact, the way I phrased it 'Using my
own rc stamp after discussion with Curtis.' only serves to highlight
that I hadn't cross-checked that process. Another reason to tune the
motivations in that process doc, or change it.

So the second problem is that in taking this step I appeared to step
over a deeply ingrained reflex in the team, to never do something
unilaterally. I say appeared to because I didn't - Curtis and I
completely agreed that it was release critical, and under the new DB
SQL query policy (if it applies) that is totally covered... as long as
I requested Curtis to push the button on Launchpad.

This appearance triggered nearly an hour of fairly intense
conversation on #launchpad-dev, and it concerns me that we appear to
have so little confidence in our own and each others ability to assess
things, that this was ever an issue: the branch was explicitly one the
current RM would have approved; the current RM was in the loop; the
domain expert for the area of the code affected was in complete
agreement, and yet we still got hung up about it.

I won't recapitulate this discussion - its in the channel logs, but I
think we need more eyeballs on this : either to help me understand the
actual problem - not that I did something different, but the
/problem/; or to figure out what we want here. Processes like the
release process - one that causes a pipeline stall for nearly 30
people for a week - deserve critical analysis, because if they stop
giving us lots of benefit, they end up just costing us all.

The release-features-when-they-are-done process will not *immediately*
move release week, so we're still going to have this process, or
something closely related to it, going on on an ongoing basis until we
really eliminate downtime. So I think its worth thinking pretty deeply
about this.

-Rob