launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Wed, Aug 4, 2010 at 7:33 PM, Stuart Bishop
<stuart.bishop@xxxxxxxxxxxxx> wrote:
> On Wed, Aug 4, 2010 at 9:46 AM, Robert Collins
> <robert.collins@xxxxxxxxxxxxx> wrote:
>
>> Doing this will immediately close a half-dozen bugs, and focus our
>> timeout and performance efforts closer to the actual source of our
>> problems.
>
> Which bugs? The only open ones I'm aware of are feature requests which
> I don't think are on anyone's radar to address.

As Martin says, milestone and bugs pages are caching inappropriately.
https://bugs.edge.launchpad.net/launchpad/+bug/601051 was originally
filed in a pretty general way, got refreshed today.
The 'hot bugs' list in a bug context is cached and is jarring (because
a common thing to do with a hot bug is to triage it).
https://bugs.edge.launchpad.net/malone/+bug/602936

This bug proposes using memcached to address cold cache problems by
preloading a lot of batch sizes into it (not a great strategy IMO).
https://bugs.edge.launchpad.net/rosetta/+bug/534203

> I'm not sure why turning the facility off will help peoples focus -
> memcached was never about stopping timeouts and this has been
> repeatedly stressed.

That message and developer actions are mismatched.

> Using the facility will slow down initial loads,
> and may improve the median page load time. It won't fix timeouts. It
> may reduce the frequency of timeouts. The memcached infrastructure is
> about improving scalability and overall performance, and throwing away
> our 24% hit rate seems rather pointless (better rate than I was
> expecting actually, but I guess the places it is being used have been
> specifically targeted). The same rationales for turning off would be
> applied to turning off Squid, which fills pretty much the same role
> but more so by caching 100% of our unauthorized access.

Caching anonymous stuff is rather easier, because anonymous queries
can't create a situation with stale data - if the same user has a
logged in and an unlogged in account they'll detect the difference,
but not otherwise.

> Turning it off is also rather problematic for foundations scalability
> and performance work - it is the most likely replacement for OAuth
> nonce and Session storage.

So, turning it off may be over-broad; removing it where it is being
abused is a more accurate way of stating my intention; I'd be happy if
we turned it off before the rollout, landed a patch to rollback the
inappropriate uses, and then turned it back on for appropriate uses. I
certainly don't want to impede scalability and performance work. Using
memcache for that does raise some concerns though: wouldn't a system
outage of memcache (like say, a kernel vulnerability forcing a reboot)
cause all users to have to log in again? We can take this to a
different thread.

>> In the future, I would like to be able to turn memcached back on in a
>> more scaling role, which is what it is designed for and great at.

-Rob