launchpad-reviewers team mailing list archive

[Colin Watson <mp+409555@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:speed-up-traceback into launchpad:master.

Commit message:
Suppress unnecessary linecache check in tracebacks

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/409555

While looking for something else in `strace` output, I happened to notice lots of `stat` calls for Python source files.  It turns out that the `traceback` module uses `linecache` to cache the contents of source files, and it checks that the cache is up to date every time it generates a traceback.  In typical Python code this isn't a big deal, but Launchpad generates a lot of tracebacks: for instance, every query while a request timeline is active causes a traceback to be stored in the timeline.  This results in a lot of system call noise.

Since production deployments restart processes, and since gunicorn is configured to auto-reload in development, it seems reasonably safe to suppress these cache checks.  I don't have accurate measurements of how much time it saves, but it certainly makes `strace` output less noisy.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:speed-up-traceback into launchpad:master.

diff --git a/lib/lp_sitecustomize.py b/lib/lp_sitecustomize.py
index 1325c1d..1f75c6f 100644
--- a/lib/lp_sitecustomize.py
+++ b/lib/lp_sitecustomize.py
@@ -6,9 +6,11 @@
 
 from collections import defaultdict
 import itertools
+import linecache
 import logging
 import os
 import sys
+import traceback
 import warnings
 
 from twisted.internet.defer import (
@@ -166,6 +168,23 @@ def customize_logger():
     silence_swiftclient_logger()
 
 
+def speed_up_traceback():
+    # The traceback module calls linecache.checkcache for each traceback to
+    # make sure line numbers of source files are up to date.  We generate
+    # tracebacks rather frequently in timelines; source files are unlikely
+    # to change over the course of a Launchpad process, especially since
+    # gunicorn auto-reloads in development configurations; and a couple of
+    # milliseconds per timeline action adds up.  Suppress these checks.
+    class UncheckedLinecache:
+        def checkcache(self, filename=None):
+            pass
+
+        def __getattr__(self, name):
+            return getattr(linecache, name)
+
+    traceback.linecache = UncheckedLinecache()
+
+
 def main(instance_name=None):
     # This is called by _pythonpath.py and by the standard Launchpad script
     # preamble (see LPScriptWriter in setup.py).  The instance name is sent
@@ -184,6 +203,7 @@ def main(instance_name=None):
     silence_warnings()
     customize_logger()
     set_default_openid_fetcher()
+    speed_up_traceback()
     checker.BasicTypes.update({defaultdict: checker.NoProxy})
     checker.BasicTypes.update({Deferred: checker.NoProxy})
     checker.BasicTypes.update({DeferredList: checker.NoProxy})