launchpad-dev team mailing list archive

[Deryck Hodge <deryck.hodge@xxxxxxxxxxxxx>]

Hi, all.

We just had to abort the nodowntime deploy today due to a late
discovered bad revision.  So we have ourselves in a bit of a bad state
today in terms of deployments, qa, and landings.  It's so serious that
wgrant is wanting us to stop landing stuff until it's done.  Being
that we have no easy way to enforce blocking merges to stable now, I'm
not sure it's worth worrying about stopping landings.  But I do want
to make sure we're all clear on what's going on, what is bad about
this, and what we need to do to fix it.

Why so serious?
-----------------------

We've had a series of bad revisions hit stable and subsequent
rollbacks.  Because we've done this for 3 or 4 bad revs over the last
6 days, we're now blocking 50 or so revisions.  This will be a large
deployment when it happens.  While this has it's own risks, we also
risk making it worse if we have another round of bad revs and
rollbacks.  We can get stuck in a vicious cycle of marking stuff bad,
rolling back, and then finally discovering new issues with the large
deploy.

What do we need to do?
-----------------------------------

In short, get a deploy done ASAP.

Henning has landed a rollback for r13766 in r13777.  As soon as this
merges to stable, we need to get QA caught up to r13777 (about 4 revs
it looks like) and get a deployment done.  While we can't block
landings, let's don't try to make this rollout any more inclusive.
Let's make Henning's r13777 the deploy target if we can.  Anything
that lands after Henning's r13777 can go in another nodowntime deploy.
 I'll help with qa or ping people about qa once the qa report is
updated.

If you want to land something, check on the state of the deployment
before sending something off to land. :)

If anyone has any concerns or objections about this plan, please speak up.


Cheers,
deryck


-- 
Deryck Hodge
https://launchpad.net/~deryck
http://www.devurandom.org/