launchpad-dev team mailing list archive

[Martin Pool <mbp@xxxxxxxxxxxxx>]

On 5 August 2010 05:27, Jeroen Vermeulen <jtv@xxxxxxxxxxxxx> wrote:
> On 2010-08-04 19:10, Robert Collins wrote:
>
>> For clarity, I'm proposing removing all the uses of it that users are
>> noticing (and thus complaining about) - not removing the facility from
>> the datacentre: memcached is a good tool and its good that we have it
>> available.
>
> If the main problem is "user changes something but doesn't see the change
> reflected in the page," that's the same problem we have with replication
> lag.  Couldn't we solve that in the same way, by having a user bypass
> memcached for a while after a POST?

We could.  It would be a fairly cute way to solve the "but I thought I
just changed that?" bug, but it's not a a total solution: "but Jeroen,
I thought you said you fixed that?"  On the whole I would still call
it a bandaid.

> (Yeah, I know, slower pages.  But set the duration of that special state to
> 10 seconds and have memcached invalidate after 10 seconds, and we'll still
> get effective "slashdot protection" while hiding all unwanted caching
> effects at minimal cost to the user.)

One thing we could do is to use feature flags to turn on or off TAL
caching, so that we can make the correctness/throughput tradeoff
dynamically when we're being slashdotted.  (Again, flickr etc
apparently use this technique.)

It seems to me that now we have TAL caches sufficiently widely used
that the cache is full and causing evictions, we have the chance to
get some data about how well it works.  We do know caching causes some
user distress.  Can we do an A:B test by turning it off and observing:

 * median load time (at least for interesting pages)
 * app and db server load
 * OOPS rate

I think it would be fascinating to see whether/how these things
change, so we can know if it does actually help with capacity.  I
guess they won't tell us how our overload behaviour changes, but they
may be suggestive.

-- 
Martin