launchpad-dev team mailing list archive

[Graham Binns <graham@xxxxxxxxxxxxx>]

On 14 September 2010 10:55, Jonathan Lange <jml@xxxxxxxxxxxxx> wrote:
> Hello,
>
> I've noticed that negotiating the downtime for Launchpad rollouts is
> becoming increasingly tricky.
>
> So I can be clear when asked,
>  * what's the downtime for rollout now?
>  * are we doing anything to reduce it?
>  * when are we expecting to have zero downtime for rollout?
>
> I'll put the answer on a wiki somewhere once the thread winds up.

Bringing this to the thread from the discussion on the TL call today:

One problem we had with scheduling the downtime for 10.09 was LOSA
availability. Not so much in having a LOSA available for the release
itself, but in terms of picking the week that we rolled out. As it
stood, Week 4 looked like this:

 * Monday - Labor Day in the US - All US LOSAs on leave; Tom H on leave also
 * Tuesday - Tom H on leave
 * Wednesday - Tom H on leave

That meant that for Monday we had no LOSA coverage between ~08:00 UTC
and 00:00 UTC the following morning. For Tuesday and Wednesday we had
no LOSA coverage between 08:00 and 13:00 UTC and we were of course
sharing the one LOSA that came online at 13:00 (Chex) with Landscape.

Oh, and staging was down. A lot.

I think we need to be more careful about scheduling downtime in a week
when there aren't many LOSAs available leading up to the rollout. As I
said, the rollout itself is fine - all the LOSAs are more than capable
of handling it - but the QA period leading up to it is just as
important.

-- 
Graham Binns | PGP Key: EC66FA7D
http://launchpad.net/~gmb