launchpad-dev team mailing list archive

[Free Ekanayaka <free.ekanayaka@xxxxxxxxxxxxx>]

Hey,

|--==> On Thu, 12 Aug 2010 15:34:30 +1200, Robert Collins <robert.collins@xxxxxxxxxxxxx> said:

  [...]

  RC> This makes it very hard for folk looking at a template, or a python
  RC> function to reason about its performance without knowing the
  RC> underlying data model very well : this raises the bar for
  RC> contributors, makes performance tuning more tricky, and is generally a
  RC> negative thing.

  RC> I have some long term ideas about this: broadly, to separate out
  RC> 'looks like python' from 'looks like it does DB access' so that its
  RC> obvious when db access is going to happen, and structure things so
  RC> that:
  RC>  - if it looks cheap, it is cheap - always
  RC>  - no database access is needed during template rendering (as a safety
  RC> measure for the first point : templates look cheap always)
  RC>  - set based, fast behaviour is easy and the default

That's sound very good to me. Especially the db-less templates, it looks
a sane design principle to me, and I believe it would also make both
templates and views more testable on their own (they'd be somehow less
coupled).

  RC> Can Storm help us?

Yes, and I believe that's the proper layer to address (at least part) of
the specific db-fetching issue you raise at the beginning of your post.

Ideally the ORM shouldn't prevent you from doing anything you could do
with direct SQL, where usually you want to load as much as possible with
as few queries as possible (though this might be true in some cases). On
the contrary the ORM should make it easy to do so, or at least less
tedious than it would be with raw SQL.

In general I tend to think to the ORM as convenient way to generate SQL
queries and structure their output, but of course one should ideally
always have a clear idea of what the generated SQL look like.

  RC> Free has an eager loading branch -
  RC> http://bazaar.launchpad.net/~free.ekanayaka/storm/eager-loading. What
  RC> this does is a storm-connected version of our prejoins. I believe that
  RC> like our prejoins its opt-in when you need it (good), but its also
  RC> very unsophisticated. Specifically it isn't constrained (so it cannot
  RC> model is_ubuntu_coc_signer),

I'm not sure to fully understand this, would you elaborate on this
specific case?

  RC> and it only (appears) to handle a single link.

I assume here you mean it handles only Reference's as opposed to
ReferenceSet's.

Actually I think that the low-level changes that the branch introduces
allows to support eager-loading not only for Reference's but also for
both direct and indirect ReferenceSet's (meaning 1-to-many or
many-to-many, IOW with and without a link table).

However I didn't yet find a good way to expose this eager-loading
options for ReferenceSet's in the Storm API, like how to control when
and how eager-loading should be used in those cases. If you're
interested to know more about that, please just ping me. Eventually I'd
really like to see Storm have a good support for eager-loading, in one
way or another.

Cheers,

Free