launchpad-dev team mailing list archive

[John Arbash Meinel <john@xxxxxxxxxxxxxxxxx>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


...

>>
>> Diff is fast, getting the commit message is fast, getting file content is
>> fast. It does seem like there is a lot we could be getting from bzr that
>> wouldn't negatively impact render times.
> 
> Are they really? Pick a random, cold cache launchpad (itself) branch.
> Get me the diff from tip to -1. Is that subsecond for the whole
> operation? yes? It might be fast; now try on a linux branch.
> 
> We may have *hot* cache operations fast subjectively, but that doesn't
> mean we're at all ready to claim its fast when thrown random data
> sets. Our data store massively exceeds main memory on either the
> storage server or web servers for loggerhead: we have a significant
> optimise-and-iterate job ahead of us to make it fast : and its one we
> should do.
> 
> bzr and loggerhead have made huge improvements in performance, but web
> services are looking at a different scale again, and I'm not at all
> convinced we're there yet (OTOH I don't think git or any of the other
> DVCSs are there *either* - cold cache performance is hard).
> 
> For instance, getting the commit message for a revision from
> postgresql is (at the moment) 8ms *cold*, 0.8ms *hot*.
> 
> -Rob

TIMEIT "b = branch.Branch.open('launchpad/devel'); b.lock_read()\
 r = b.last_revision(); rev = b.repository.get_revision(r);\
 r.message; b.unlock()"

Hot, is 15ms. If I change it to:

TIMEIT -s "b = branch.Branch.open('launchpad/devel');" \
  "b.lock_read(); r = b.last_revision();\
   rev = b.repository.get_revision(r); r.message; b.unlock()"

It is 0.361ms to lock + read the content + unlock. So most of that time
is spent in opening the branch.

TIMEIT "call('echo 3 | sudo tee -a /proc/sys/vm/drop_caches',
shell=True, stdout=PIPE)"
24ms

TIMEIT "call...; Branch.open(); ..."
49.6ms

TIMEIT "call...; b.repository.get_revision(r)"
26.9ms

So that looks like:

1) 25.6ms cold to Branch.open() + b.last_revision() +
               b.repository.get_revsion()
2) 2.9ms   cold to b.repository.get_revision()
3) 15ms    hot to Branch.open()...
4) 0.361ms hot to b.repository.get_revision()

Now, I believe you said that XMLRPC is in the 50ms range on the local
system, which is a pre-requisite for loggerhead looking up the path to a
branch, and all sorts of extra overhead. (HTTP Branch.open() vs
local-disk Branch.open()

$ TIMEIT "b = branch.Branch.open('devel')
b.lock_read()
r = b.last_revision()
p = b.repository.get_parent_map([r])[r][0]
rt, pt = b.repository.revision_trees([r,p])
l = list(rt.iter_changes(pt))
b.unlock()"
100 loops, best of 3: 10.4 msec per loop


Now, clearly something very strange is going on, because
repo.get_revision() was up at 15ms, but iter_changes is down at 10.4ms.

$ TIMEIT "call('echo 3 | sudo tee -a /proc/sys/vm/drop_caches',
shell=True, stdout=PIPE)
b = branch.Branch.open('devel')
b.lock_read()
r = b.last_revision()
p = b.repository.get_parent_map([r])[r][0]
rt, pt = b.repository.revision_trees([r,p])
l = list(rt.iter_changes(pt))
b.unlock()"
10 loops, best of 3: 44.9 msec per loop

So that is
1) 31ms cold to do iter_changes,
2) 10ms hot to do iter_changes.

That is pretty far underneath your 1s threshold, even cold. Though I'm
leaving out all of the ugly XMLRPC stuff that we have in production.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk4J6owACgkQJdeBCYSNAAMdxQCgvoJG0jxviOEkPIWLtbf3wf1i
JDkAn1Hzf6JSPfjMPTjeVX2pMPnlTBvz
=nE5a
-----END PGP SIGNATURE-----