[Launchpad-users] Edge rollouts at the weekend

[James Westby <jw+debian@xxxxxxxxxxxxxxx>]

Hi,

I woke on Saturday to discover that the edge rollout had apparently
failed halfway through and left at least one machine on an old revision.
This plays havoc with the API, and meant that my scripts could not
work all weekend, with nothing I could do to fix it (with edge the
only option for running scripts currently).

Apparently things got worse on Sunday, with the webapp playing up
as well.

As there is no LOSA available at the weekend (which I have zero
problem with, they deserve their weekends), I'm not sure that
there should be automated rollouts on Saturday and Sunday mornings,
as if they go wrong it makes life hard for those that wish to
use LP at the weekend.

As it is edge, redirection can be disabled (except for API uses),
however, if people feel like they have to disable edge redirection
every two hours every weekend, then they are likely to leave the
beta test team to avoid the problem altogether.

There is a QA benefit to deploying the latest code every day,
and that is valuable, however, delaying some of that feedback for
a day or two by not rolling out at the weekend wouldn't be a big
loss as there are few LP developers around at the weekend anyway,
and would increase the productivity of a valuable resource, your
beta testers.

Alternatively there may be work underway to make the automated
rollouts more robust, which may make the recent problems with
the edge servers being inconsistent go away, leaving just the
occaisional show stopper bug introduced by a code change, which
would make the problem less frequent.

Thanks,

James