launchpad-dev team mailing list archive

[Danilo Šegan <danilo@xxxxxxxxxxxxx>]

Hi Robert,

I think it's great that you are focusing on performance.  And while
database used to be the main contention point in the past, my feeling is
that it is not so anymore.  I.e. we know how to fix DB performance
issues, and many remain simply because of the lack of time to go through
all of them.  I might, of course, be mistaken, and people don't really
have the same experience tackling performance problems like we do in
Translations. :)

The discussion you are starting is very useful, because we should
ultimately be able to design APIs we like while not sacrificing
performance.  Today, in Launchpad, you usually have to come up with ugly
APIs to accommodate performance solutions you are employing.

У чет, 12. 08 2010. у 15:34 +1200, Robert Collins пише:

> I have some long term ideas about this: broadly, to separate out
> 'looks like python' from 'looks like it does DB access' so that its
> obvious when db access is going to happen, and structure things so
> that:
>  - if it looks cheap, it is cheap - always
>  - no database access is needed during template rendering (as a safety
> measure for the first point : templates look cheap always)
>  - set based, fast behaviour is easy and the default

It would be very nice to come up with a sane approach for these kinds of
things which doesn't include methods named like
"getPOFilesAndPOTemplatesAndPersons", which is roughly the manual
approach we are taking today (i.e. getting a ResultSet containing all
objects we care about in one single query).  My opinion is that we could
just start with Free's work on Storm (cool stuff, Free!), expand it so
it works well even with cases like ubuntu_coc (though, it seems to me it
already supports a lot of it, and it does support more than one argument
prejoin), and just come up with a coding convention in Launchpad to
express pre-joins in a generic way.  If that means starting with
something simple like model API methods defined like:

  def getPOTemplates(populate=[POTemplate.sourcepackagename])

that's probably good enough for now (i.e. standardize on the argument
name, whatever it is — "populate" is just an example — and always make
it a column list or something).


However, even when we use the optimal DB path, we have problems with the
rendering time itself.  Check out
https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1685EB1355
for an example.  That is 275ms of SQL time (and no, it's not a bug in
the OOPS tools: that is the SQL time), and ~17s of rendering time, and
we believe fmt:url for each object is one of big culprits there (well,
removing it cuts the rendering time to 5s :).  However, to avoid doing
it, we'd have to do something like
getPOTemplatesAndSourcePackageIfNeeded(), so generalizing "prejoins"
using Storm would help here as well, but ultimately, we need to fix
fmt:url to not take this long.

Granted, this is rendering ~2000 objects and doing fmt:url on each one
of them, and we can argue that it's not a good idea (there are reasons
why we are putting it all on one page).  We knew that DB could handle it
just fine, but we didn't expect the Python code to be this slow (to be
honest, initial version of this did even more fmt:urls and even used
menu:navigation which caused the page rendering time to go into hundreds
of seconds).

I know there are plans to fix/look into many of these things (I've also
talked it over with Gary during the Epic), however, I somehow feel that
you are neglecting this entire area of Launchpad performance.


I.e. even if we went and fixed the DB side of things for pages like

  https://lp-oops.canonical.com/oops.py/?oopsid=OOPS-1685EC1691

we'd still be left with >5s of generation time on the appserver.

(If it was only pages like these, I would just say "let's worry about it
later", but because of the pages like the first one, this is an actual
problem that we hit right after we resolve all the DB issues on a single
page)

Cheers,
Danilo