launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

This one time, at weekly catchup call, Francis and I spoke about this
plan ... :)

We both think that its going to take significant time to get all the
pieces in place, so we're proposing to make this whole thing even more
incrementally adoptable.

So - first up, a 'daily staging' environment, which is RT 40482 qa
with the stable branch and production schema.
This will let us start QA'ing what we rollout on a daily basis (vs the
monthly cycle for db-stable). QA seem to be pretty much ready for this
- can you confirm that this is in fact the case?

We'll also want to fix RT 40685 deploy icing to apache before updating
appservers, because rollouts to appservers currently cause a few
glitches.

When the branch 'stable' is completely QA'd, a rollout of *just the
appservers* will be triggered (initially by asking a LOSA to hit the
button), which includes stable getting propogated to the 'production'
branch as normal.

Once these two things are done and we're successfully executing this
rollout pattern, we can just get rid of the 'edge' config, 'edge'
appservers and so on. We'll keep the edge DNS entry, with an apache
config that simply redirects to the same url on production. is_edge in
our codebase will start returning False *everywhere*. We discussed an
alternative, where we kept edge working as a DNS alias for production,
and turned on a feature scope for edge; but we felt that the gains
were marginal (users could opt in/out of *one* scope - the edge scope
- directly) and something we want a better answer for anyway (users
should be able to opt into / out of things on a more granular basis -
e.g. participate in A/B testing and choose to get the original format
page back because they don't like the new one). With that level of
control over scopes, an implicit edge scope started to feel positively
primordial, and it has lots of costs - an ongoing redirect to edge
causes performance issues in new browser sessions, we'd have more QA
work to do, because its a global scope that would be affecting every
page, and so forth.

This eliminates some complexity from the production environment
immediately : rather than 1 revision for all backend servers + some
appservers, and a different revision for the edge appservers, we will
have one revision for all backend servers and one revision for all the
appservers.

It also means that changes which you're experimenting with should be
hidden by feature flags. These are available in DB-stable now - so
please do start using them asap for things you are doing in db-devel;
we'll need to develop some proficiency with it (and there are still
significant concerns Curtis raised in an earlier thread that we need
to put in place preventative measures for :- the sooner we have
concrete data on them the easier is it change our path accordingly.

The next steps after that are to iterate on the reliability of doing
backend rollouts - cronscripts, single points of failure etc - until
we have just a single revision everywhere.

At that point, we can start working on the database patch deployment
story - but expect that to be 2-3 months away for now.

I will (honestly!) update the wiki page tomorrow, but I wanted to get
this proposal out to you guys asap, so you can tell me how its broken!

-Rob