launchpad-reviewers team mailing list archive

[John A Meinel <john@xxxxxxxxxxxxxxxxx>]

John A Meinel has proposed merging lp:~jameinel/launchpad/loggerhead-clear-cache into lp:launchpad.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~jameinel/launchpad/loggerhead-clear-cache/+merge/119793

This is a small tweak for the Launchpad Loggerhead app that drive bazaar.launchpad.net.

Specifically, every request to loggerhead gets a new bzrlib.branch.Branch object created. Which then loads data for all the indexes, etc. A lot of this gets cached on the assumption that the object will get re-used. However, these objects are *not* shared between Loggerhead requests. It is a known issue that they also don't go away by simple ref-count (they require the garbage collector to run to remove circular references). 

So this is just helping things along. Once we've finished an HTTP request, directly request that caches at least get cleared out.


In doing a test of just spidering of the launchpad codebase using bazaar.launchpad.dev on my local machine. This dropped the peak memory of 'start-loggerhead.py' down from 500+MB down to around 130-200MB.

A different option would be to look into changing how branches are opened, and possibly caching them. It would theoretically help performance a lot, we wouldn't be re-reading all the same content over and over. However at this point, I'd rather drop a cache than add a new one.

-- 
https://code.launchpad.net/~jameinel/launchpad/loggerhead-clear-cache/+merge/119793
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jameinel/launchpad/loggerhead-clear-cache into lp:launchpad.

=== modified file 'lib/launchpad_loggerhead/app.py'
--- lib/launchpad_loggerhead/app.py	2012-06-29 08:40:05 +0000
+++ lib/launchpad_loggerhead/app.py	2012-08-15 21:05:07 +0000
@@ -286,6 +286,12 @@
                     served_url=None, private=private)
                 return view.app(environ, start_response)
             finally:
+                bzr_branch.repository.revisions.clear_cache()
+                bzr_branch.repository.signatures.clear_cache()
+                bzr_branch.repository.inventories.clear_cache()
+                if bzr_branch.repository.chk_bytes is not None:
+                    bzr_branch.repository.chk_bytes.clear_cache()
+                bzr_branch.repository.texts.clear_cache()
                 bzr_branch.unlock()
         finally:
             lp_server.stop_server()