launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Wed, Sep 15, 2010 at 12:46 AM, Curtis Hovey
<curtis.hovey@xxxxxxxxxxxxx> wrote:
> On Tue, 2010-09-14 at 11:41 +0100, Tom Haddon wrote:
>> Might be more reliable but less accurate :) We estimate the downtime
>> based on how long the last update took on staging, and then
>> multiplying
>> by a factor that seems to have accurately reflected the difference in
>> time between staging and production (with a little padding). We could
>> only commit to 90 mins if we refused to rollout any DB updates that
>> took
>> longer than a certain period of time on staging.
>
> Staging restore times trend up, so we are always talking about
> increasing time for a rollout. We will continue to do schema development
> after the featureflag is complete. What we cannot see is the staging
> restore time verses the real time--maybe that is pointless because there
> are other rollout incidents that increased the rollout.

Staging restore times as a whole are a poor surrogate as already discussed.

The point I am making is that unless we decide *how much downtime we
will tolerate*, we'll always have reasons to do more.

So I'm proposing:
 - a 90m budget.
 - if we can't do it in that timeframe, we don't do it.

We *will have to* innovate and address various issues to stick to
this, but 90m of time is actually a lot of lost time to the many
thousands of users we have in every timezone. We could spend a week
with the whole team working on something to make the upgrade faster,
and still be spending less time than our users are losing, when we're
down.

-Rob