← Back to team overview

launchpad-dev team mailing list archive

Using feature flags to ameliorate timeouts this release

 

So we have a release coming up. I suspect I'll be sound asleep when it happens.

We also are deploying new DB schemas which (as usual) have a risk of
causing timeouts.

This release will make the hard_timeout flag available.

I'd like to encourage folk to look at
https://devpad.canonical.com/~lpqateam/lpnet-oops.html (which I
understand to be about 1 hour delayed) and see what pages are timing
out. Then add a feature rule (https://launchpad.net/+feature-rules) to
bump those pages back up to a higher figure.

E.g.
hard_timeout pageid:BugTask:+index 1 15000    (*)
would set the BugTask:+index page timeout to 15 seconds.

The rules for changes to feature flags are as for any production
change - get an ack from a teamlead, jml, francis or myself.

It would be great (but not mandatory) to ensure there is a bug on
https://bugs.edge.launchpad.net/launchpad-project/+bugs?field.tag=timeout
for any rules you need to add - I'm basically expecting to catch all
the pg8.4 regressions here too.

I would add them now, but we can't really trap them at the moment -
the hard_timeout flag only affects edge until the rollout.

I will be diving into doing this after the release, but as I say -
I'll be snoozing, at least initially ;).

-Rob

*: Anatomy of a rule:
FLAG SCOPE PRIORITY VALUE
so in the example I gave:
FLAG=hard_timeout
SCOPE=pageid:BugTask:+index
PRIORITY=1 (pick a unique number per FLAG), highest priority matching SCOPE wins
VALUE=timeout_in_milliseconds