launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

On Wed, Aug 11, 2010 at 4:27 PM, Michael Hudson
<michael.hudson@xxxxxxxxxxxxx> wrote:
> On Wed, 11 Aug 2010 15:02:13 +1200, Robert Collins <robert.collins@xxxxxxxxxxxxx> wrote:
>
>> The latter are trickier because we want to get all the related data we
>> want at once, no more than needed, and no less. If we get more things
>> go slower, and if we get less we scale poorly as the page gets busier
>> (e.g. more builds). The pattern I've tried in registry with
>> _all_members should help considerably, once we get the cache coherency
>> stuff sorted (which strictly speaking only really affects the test
>> suite : production clears storm caches between requests, and so caches
>> on model objects have no lifetime in the actual server).
>
> What's the issue here, generally speaking?

I don't know enough to speak generally.  I can tell you about a
specific problem.

> It's possible to arrange for
> caches to be cleared on many boundaries -- for example, the per-request
> security policy cache is cleared when the principal changes, which
> happens during login.  Clearing all @cachedpropertys like this might not
> scale very well, but otoh there might be something we can do.

Sure.

So there are two caches per model object [at a baseline level]

*) cachedproperty. This is stored in the model object, as
_attributename_cached attributes.
*) storm data to determine changes made when flushing a change.

And there is a cache /of/ model objects: the storm per-thread cache.

Cache invalidation for the first: doesn't exist, unless written adhoc
& manually.
Cache invalidation for the second is automatic on flush and attribute
reset when a query hits the db.
Cache invalidation for the third happens:
 - in production at the end of every request: we throw away the entire cache
 - in testing, never.

This means that things like the cachedproperties I added blow up in
testing a lot, and rarely-if-ever in production.

We can use __storm_invalidation__ or whatever the hook is to tell
storm to invalidate the cached properties, and I'll do that : what I'd
like to do long term is:
 - remove all manual caching
 - remove the entire overhead of storm caching - *quite a lot* of code
runs on every setattr & on so on. It does have a C accelerator but it
is 99% of the time waste: we are write-light, read-heavy.
 - discard all objects at the end of every transaction
 - explicitly track updated fields, and only updated fields for updates.

-Rob