launchpad-dev team mailing list archive

[Jamu Kakar <jkakar@xxxxxxxx>]

Hi Rob,

On Wed, Aug 11, 2010 at 8:34 PM, Robert Collins
<robert.collins@xxxxxxxxxxxxx> wrote:
> I have some long term ideas about this: broadly, to separate out
> 'looks like python' from 'looks like it does DB access' so that its
> obvious when db access is going to happen, and structure things so
> that:
>  - if it looks cheap, it is cheap - always
>  - no database access is needed during template rendering (as a safety
> measure for the first point : templates look cheap always)
>  - set based, fast behaviour is easy and the default

+1!

> Free has an eager loading branch -
> http://bazaar.launchpad.net/~free.ekanayaka/storm/eager-loading. What
> this does is a storm-connected version of our prejoins. I believe that
> like our prejoins its opt-in when you need it (good), but its also
> very unsophisticated. Specifically it isn't constrained (so it cannot
> model is_ubuntu_coc_signer), and it only (appears) to handle a single
> link. That said, its not finished yet, and perhaps we can make
> prejoins a bit nicer with it.
>
> A related problem with prejoins is that they only help with
> *reference* lookups, not ReferenceSet :  a ReferenceSet will query the
> DB again, a Reference that has been prejoined won't (because the
> primary key is in the storm object cache). This means that when you
> have many rows with many different base objects, prejoins will help,
> but they won't ever be the full story.
>
> I think Storm will be of great assistance in the long term stuff I
> mention above, but in the short term we want to achieve something
> which a complete solution to will put considerable stress on storm -
> *all* our derived data lookups that trigger in web pages or API's need
> to be expressable for a depend-on-storm solution to be feasible. Using
> storm features where we can obviously makes sense.

It'd be nice for Storm to be more helpful.  Free's branch is a good
start, but I think it's important to be cognisant that much of the
magic in an ORM are beautifully crafted bandaids to work around the
problems they introduce.  IOW, I'm glad you're talking about
different ways to think about the problem.

> I'm working on an automatic invalidation facility for our model
> classes, so that they will invalidate when storm invalidates an
> object.

If you provide a __storm_invalidated__ method (taking no parameters)
on your Storm objects it will be invoked when Storm invalidates the
object.  We use this in Landscape all over the place to invalidate
model-level caches and it's been working well for us.

Thanks,
J.