launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

So, I'm looking at milestones at Francis suggestion - the previous
changes with registry Persons to introduce an extended idiom for
caching data in model objects and prepopulating such caches is pretty
nice, but we don't yet know if its broadly applicable or scalable in a
code-reuse sense.

Milestones have the nice (or horrid :P) property that they touch many
things, so we're going to push this idiom's flexability and ease of
use to make it work there, IF milestones are suffering death by sql.

Bug https://bugs.edge.launchpad.net/launchpad-registry/+bug/447418 is
the relevant bug, and a quick check of an oops shows serious sql pain:

SQL time: 7505 ms
Non-sql time: 7849 ms
Total time: 15354 ms
Statement Count: 390

In particular, one query is looking up private bug subscriptions, one
query per bug: this is classic death by sql, one that prejoins won't
help for, but which prepopulating caches will - so I'm setting out to
have an explore.

I've got a test already, which shows 41 queries are needed to show an
empty, new milestone with one private bug attached to it. Adding a
second private bug takes the query count to 42 : exactly the behaviour
seen on production, so a fix there should (if we test that the
combined queries perform no worse than the current individual ones)
make things better.

We can test the combined lookup times on staging, and head on from there.

I'd be *delighted* to pair on/mentor an application of the
cachedproperty prepopulation idiom with anyone interested in applying
it: just email or grab me on IRC. I'm particularly interested in
finding out other patterns we have that *prevent* it working well,
because it would be a nuisance to be encouraging something we can't
easily use :)

-Rob