launchpad-dev team mailing list archive

[Robert Collins <robert.collins@xxxxxxxxxxxxx>]

Hi Ian, yes this is a problem I see too. I am totally for separating
data mapping and domain logic. I don't remember which threads I've
mentioned this in, but the first and second Performance Tuesday
threads probably have some ramblings.

I don't think the units tests are a key aspect of it - performance of
those tests is very important to us for agility and dev throughput,
but there are more profound and significant problems that are caused
by the current structure. And, optimising the unit tests should come
*after* optimising production : if we, for instance, made all our
pages twice as fast, we can reasonably expect the test suite to get
faster (because the test suite is exercising code paths that are now
twice as fast). But the reverse isn't true: making the test suite
faster is much less likely to help production performance [unless one
makes the test suite faster by making production code paths faster].

A primary problem I see is that we have no high level language [other
than object traversal] to describe how much of the object graph we
wish to retrieve. E.g. 'Milestone X, all specs & bugs, the product and
project group, all people associated with X, or in key roles on any
related item, branding for any teams found'.

We have a bunch of adhoc stabs in this direction; things like the
recent GenericCollection, and my  recent refactoring of
Person.all_members. None of them are hugely satisfactory *yet* : you
cannot combine different collections reliably (yet - that might be a
nice thing to work towards); and its very ad hoc.

Because we have no high level language to describe what a page will
need, domain objects end up requesting more of the object graph in
method calls and attribute access (because, until -very- recently, we
didn't actually build an object graph, rather we always went back to
SQL every time).

As a result pages often end up issuing hundreds or thousands of
queries in a very inefficient manner.

And so things are slow.

I see this as a limitation of Storm, our ORM : other ORM's (both in
and out of the python world) have an explicit, decoupled, mapping
layer. Working with Storm though, the usual idiom (at least in
Launchpad - I need to spend a couple of days with Jamu in the
Landscape code looking for inspiration) is to be expressing things at
the SQL layer.

In terms of what we should do here; I was imagining a mapping *layer*,
not necessarily DAO classes, - things like GenericCollection, and
BugTaskSet are, to me, mappers, and would fit in that layer. I'd love
to have the layer be Storm owned and driven. Being able to express -
without SQL - the cut points in object graph traversal and have a
mapping layer return a resultset for the resulting graph would be
-awesome-.

We should, I think, be able to sensibly aim for 10 queries per page -
even the busiest pages rarely have more than 10 discrete regions
showing unrelated data.

Salgado, in terms of being able to migrate incrementally, I see it as
a fairly high risk proposition today : we haven't finished the
SQLObject migration, and eager loading interactions with that are
really rather nasty : things that look like they should work don't, we
have silently failing methods (like __nonzero__), and and unknown set
of other problems.

As a strategic concern, coding for Launchpad is full of hidden
complexity. I'd like use to incrementally remove and improve on that.
Less global state, more things-on-contexts (e.g. Request.annotations),
and clearer code in the mapping/domain area would be awesome too.

To me, finishing the SQLObject migration will remove a bunch of nasty
surprises and give us a saner base to go forward with experiments on
mapping. To be clear: I'm ok with folk experimenting with different
sorts of mapping; (I don't think a parallel class structure is
necessarily all that great - in python we could be more dynamic than
that). However, I think it will be very hard to do well, safely and
incrementally until SQLObjectResultSet specifically, is purged.

-Rob