launchpad-dev team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


...

>>>        5) Deploy loggerhead 1.19 with history_db.
>>
>> What are the implications of this; how do we turn it on? Can we still
>> share caches, and how will it interact under high load?
> 
> 	Those would be questions for jam.

At the moment every branch gets its own sqlite cache file, which is then
shared between loggerhead instances (it is put somewhere based on the
branch name, etc.)

This on-disk db is then mapped into an in-memory cache of the content
(using an LRU cache, IIRC)

Having each branch have its own sqlite db means:

 1) You avoid concurrent writer issues
 2) You really don't need an sqlite db for a single
    zlib.compress(marshal(list)) blob.

However, because the existing cache is just an 'everything in the
history of the branch', there isn't any specific benefit to sharing the
cache between branches.


For history_db, it actually will perform worse under this case, because
it is optimized for sharing information between branches. So ideally you
would have a single sqlite db for each project.

The loggerhead code gets the location of the sqlite db from the
Launchpad container (it is part of the WSGI request object). So it just
means that the updated loggerhead code needs to be released in lock-step
with updated Launchpad code. But I'm pretty sure that's how it is done
today anyway.


The other step that is desired for rollout is to pre-populate some of
the bigger projects. Specifically, the time to build the full history
for emacs is about 30s on my hardware. And we'd like to avoid pushing
that out to users on first access.

Note that accessing a new branch of emacs from a shared cache drops from
10s to about 100ms.

To summarize the performance characteristics of the old vs new code.
This is done from memory, and is mostly about getting
orders-of-magnitude correct. All of this is assuming a project like
emacs, with 100k revisions, etc:

Old	New	Action
10s	30s	First request of a branch in a project that has not been
                previously seen. (slower)
10s     100ms	First request of an emacs-sized branch when the project
                has been seen, but the branch itself has not.
		This also occurs when a request comes in for a branch
                whose tip revision has changed (rev 100 => 101)
1s      100ms   Request of the tip information for a branch that has
                seen (sqlite is populated) but is not in the in-memory
                cache
20ms    100ms   Request of tip information when the sqlite is populated
                and a copy is already in memory. (The in-memory cache
                has been removed in favor of just history-db.)

20ms    1s?     Request information about old history (rev 1) when the
                data would otherwise already be cached in memory. This
                is seen on pages like the files tree, since the
                last-modified revision for many directories is rev 1.
                There *is* an in memory cache of this specific
                information in the history-db version of loggerhead.
                Though the cache is significantly smaller in scope and
                doesn't attempt to cache information about all
                revisions in the whole ancestry.



The individual sqlite cache files can be shared. There is some concern
about multi-write. With a single loggerhead instance, I'm using an
in-memory lock to ensure that this never happens. You can do some tricks
with sqlite dbs so that the writer almost never blocks readers. Namely,
it will block them if its in progress cache overflows, but you can set
that up to 200MB and fit a full update of all of emacs in the cache.

The multi-writer case comes into play if 2 people hit previously unseen
branches of a project which hits different loggerhead instances. We
could work around this with:
 a) A separate on-disk lock that blocks the writer
 b) A simple "retry after X ms" if we hit a collision trying to "START"
    a transaction.


The new code isn't strictly a win under all possible queries. But in the
case of *common* requests it can be a significant win. And
pre-populating the sqlite db removes the primary slowdown of the new code.

Also note that one of the ideas with moving BranchRevision to using
history-db like tables in the postgres db, is that then the branch
scanner can do all the updating of the tables, and nothing needs to be
done on-the-fly by loggerhead.

John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1AhB8ACgkQJdeBCYSNAANv0wCbBFrsfftELJr0+U1gyefqDebj
De4AoNWwlSe+oqgj7oQFZ0h4dqTcP0oc
=pGEd
-----END PGP SIGNATURE-----