launchpad-dev team mailing list archive

[Robert Collins <robertc@xxxxxxxxxxxxxxxxx>]

On Sun, May 8, 2011 at 1:51 AM, Jeroen Vermeulen <jtv@xxxxxxxxxxxxx> wrote:
> If we block reference demand-loading, we start taking oopses for those
> situations.  And if we then fix those oopses by doing more eager loading, we
> may end up taking an unnecessary performance hit and pay for it in extra
> code complexity.

If we generate an OOPS it means that scenario wasn't tested. Thats
suboptimal at best. If reference loading is appropriate (I hold strong
doubts that it ever is in the webapp post traversal), then the point
doing the reference based load could reenable it temporarilty. (But
doing that implies that we could just use a single-item bulk load
too).

> Wouldn't a logging-and-monitoring approach be preferable over a
> blocking-and-fixing approach here?

I don't think this is an alternative. Its a useful technique that
solves different problems. We probably want both.

> I haven't had time to work on my Storm
> profiling branch lately, but it would let us track which references lead to
> how many new objects being fetched, coming from how many other objects.
>  With information like that we could make rational tradeoffs between eager
> and on-demand fetching.

In a year of close attention now, I've seen one case where eager
loading was a pessimisation - with commit() being slowed down by storm
pathology when 10's of K of objects are live - it has O(live) overhead
rather than O(changed). This is a bug with storm though: the eager
loading would still have made the script faster (and more consistent)
were it not for this defect.

-Rob