launchpad-dev team mailing list archive

[Tom Haddon <tom.haddon@xxxxxxxxxxxxx>]

On Tue, 2010-09-28 at 14:07 +1300, Robert Collins wrote:
> Hi, we have currently a production-stable branch which is private; it
> is maintained with CP's and merges during a cycle and discarded every
> time we bring db-stable into play.
> 
> I'd like to suggest that we make a few changes here as RFWTAD progresses.
> 
> Firstly, I think that security patches which have never been public
> are really very very rare: we can make the process for dealing with
> them a little more complex, and make the common case much simpler, for
> an overall net win.
> 
> Once we have qastaging live, we're going to be switching deployments
> to edge that haven't been QA'd, off. QA will be moving to qastaging.
> 
> At that point, if we want to, we can simply stop using
> production-devel and production-stable.
> 
> Here's how it would work.
> 
> We deploy stable rather than production-stable to servers. This would
> mean no more CP's - only cowboys and deploys.
> We shouldn't need CP's because we have the QA process Maris mailed out
> for moving things on stable into production.
> 
> And at that point, if we have a security issue we have to deploy asap;
> we'd do the following:
>  - cowboy it out there [and keep it as a cowboy on future deploys]

So this means we'd be deploying a security fix without having run the
test suite against it in a controlled environment (i.e. buildbot/PQM)?

>  - land a regular branch fixing it for good
>  - remove the cowboy when the regular branch has been incorporated
> into the main deployed codebase.
> 
> This would chop 4 hours off the time that things take to deploy,
> remove one buildbot queue and generally make the whole code->live
> story a bit simpler, at the cost of making the security-fix story more
> complex. Personally, I think that that is a net win.

>From the LOSA perspective, it's also a lot more work. It basically
requires manually applying a cowboy, keeping track of where that cowboy
is applied, disabling any auto-rollouts to that server until the cowboy
lands, and/or checking there are no cowboys applied on any servers
before doing any rollouts.

I'd propose a slight change to the above suggestion:

- Keep production-devel/production-stable (now the buildbot instances
run in the DC, there's no extra cost to doing so).
- Have an automated job that pulls frequently (or pushes immediately)
from the "approved" stable revno to production-stable
- Security fixes still go through production-devel -> production stable
and can then subsequently be landed on devel->stable after having been
rolled out.

The advantage of this is that LOSAs can *always* deploy from the tip of
production-stable. No approval is needed, and once we get to the stage
of automating deployments that becomes a *lot* easier.

Thanks, Tom

> Seeking-your-thoughts,
> Rob
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~launchpad-dev
> Post to     : launchpad-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~launchpad-dev
> More help   : https://help.launchpad.net/ListHelp