launchpad-dev team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 1/26/2011 8:41 PM, Max Kanat-Alexander wrote:
> On 01/26/2011 05:51 PM, Martin Pool wrote:
>> I want to sort out something where we can get fixes into Loggerhead
>> and deployed to Launchpad; where work done for Launchpad does not
>> gratuitously diverge from what other people run; and where work that
>> has been done into trunk is not wasted.
> 
> 	Totally, sounds very sensible.
> 
>> It seems like at the moment we have a case where trunk, though it has
>> some good work in it, isn't usable by the project's largest user, it
>> has some tests failing, it is lacking tests for some things, and seems
>> to have some regressions. 
> 
> 	That's somewhat correct. Tests are failing on the stable branch, also.
> There are no regressions from *loggerhead's* standpoint, that I know of.
> 
>> The regression, if I understand correctly,
>> is that the historydb changes make things slower in some important
>> cases,
> 
> 	No, I doubt that this is true. loggerhead already has to build an
> entire branch history and cache it when a branch is first accessed. It
> doesn't even usually store this cache persistently on disk, and it only
> stores 10 recent caches in memory. So unless there's something serious
> that I'm missing, there's no way that history_db is slower than
> non-history_db.

We can look at this, but *how* the data is stored changed. Instead of
just storing one big blob, it now splits it up into many rows across
multiple tables.

I did another round of concrete timings on my machine using 'launchpad'
as the data source. This isn't as big as emacs, but it is slow enough to
show the results I'm trying to highlight.

lp-pqm  trunk   action
6.721   19.622  First view of launchpad/devel/changes
0.156    0.072  Reload launchpad/devel/changes
0.720    0.090  Restart loggerhead, reload launchpad/devel/changes

6.345    2.208  Load launchpad/db-devel/changes (shared cache)
0.171    0.036  Reload launchpad/db-devel/changes
0.506    0.029  Restart loggerhead, reload launchpad/db-devel/changes

		pull 1 new revision to launchpad/devel/changes
8.349	 0.190	Reload launchpad/devel/changes


So concretely, building the cache from scratch is slower. It takes 20s
to build whereas it used to take 6.7s to build. All other requests are
faster, often *significantly* so.

Note especially the "1 revision change to tip of branch". Current pqm
loggerhead has to rebuild the whole history from scratch. Loggerhead
trunk only tacks on a single revision.

If we don't want to go to a shared-cache model, note that we can
pre-seed all of the caches by just grepping for what branches have been
accessed in the http log, and sending a wget for all of those pages. And
we'll still see *much* better results for people who are looking at a
branch that has been recently updated.

> 
>> and secondly that the raw view makes things insecure in some
>> important cases.
> 
> 	That is true, for Launchpad only. All of the non-Launchpad security
> cases are already handled by loggerhead itself. The Launchpad case will
> have to be handled by custom code developed especially for Launchpad--it
> will likely go into the loggerhead-launchpad project or will be a piece
> of IT configuration, rather than actually being a change to loggerhead
> itself.

I think it is also very easy to disable the raw view for launchpad until
that gets sorted out.

...


> 	There just wouldn't be any reason to update CodeBrowse to trunk if you
> weren't using history_db, because CodeBrowse already has all the other
> important features from trunk.

What about stuff like YUI 3 support?


> 
>> I guess all of us would prefer to do 1 or 2, but in the medium term,
>> absent some non-Canonical person coming along to do them, they are not
>> going to happen soon.  So we have to choose between 0 and 3, which is
>> in a way a choice of labeling or semantics.  Either could work;
>> neither is great.  For people tracking trunk it is not very nice to
>> suddenly have it jump backwards, and nor is it very nice for it to not
>> move at all while fixes are landing elsewhere.
> 
> 	Ah, well, no matter what you call it, it has to have tests that
> actually check that things aren't regressing, or the LOSAs will start
> calling it "codebounce" again. I suspect that at this very second, trunk
> and the stable branch are equally stable, but I really have no concrete
> idea beyond my personal opinion and local experience, without the tests.
> Even if you start pushing changes onto some new branch (no matter what
> you call it) there's simply too much risk that the system will become
> unstable again--particularly because you'd be working with a worse
> architecture on that stable branch than there is on trunk (which had
> some fair bit of nice refactoring).
> 
> 	-Max

Technically they still call it codebounce... (At least they did in the
IRC log last night as we sorted out issues of a nodowntime rollout
applied to one host and not to the other host.)

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1Bq3UACgkQJdeBCYSNAAPlKwCfauDWP2wBFhsTir0Ma1KDFVxu
3u8An1dcXeMpWajXeGfwNZCzVI/Q7v1h
=Xqln
-----END PGP SIGNATURE-----