yellow team mailing list archive

[Benji York <benji.york@xxxxxxxxxxxxx>]

I may be out on paternity leave starting any time now, so I'm going to
start writing these messages at the end of every day so whatever I'm
working on will be easy to pick up.

Right now I have lp:~benji/launchpad/bug-894177-2 landing.  It just got
merged into devel as revision 14476.  Once qastaging has been updated to
that version it can be QAed using the instructions at
https://code.launchpad.net/~benji/launchpad/bug-894177-2/+merge/84676

I'm going to be out Monday so if anyone has time to shepherd the 894177
branch through QA and a no-downtime deployment, I'd appreciate it.

I've also been handing over bug 808930 to Graham.  Here are the notes I
promised him on analysing the behavior of the branch scan job.

First, the built-in profiling doesn't work (--profile=FILE); it will
just profile the spawning of a subprocess that actually does all the
work.  Printing anything to stdout won't work either, since the
subprocess captures all output and sends it... somewhere.

I had good luck initially with strace to figure out roughly where time
was being spent, especially helpful are -f and -tt (see man).  Here's
one of the command-lines I used:

    strace -tt -f -T -o strace-b cronscripts/scan_branches.py

The stracing showed me that a very large number of queries were being
executed.  That lead me to try the LP_DEBUG_SQL and LP_DEBUG_SQL_EXTRA
environment variables (https://dev.launchpad.net/Debugging) but neither
worked, presumably because they weren't passed through to the
subprocess.  That lead me to hack Storm to record similar info.  Here's
the diff: http://paste.ubuntu.com/765479/

The vast majority of the DB traffic is generated by
lib/lp/codehosting/scanner/bzrsync.py so I instrumented that to give me
some rough timing info: http://paste.ubuntu.com/765475/  I haven't had
the time to analyze that much yet.

Looking at the output of the traceback dump indicated that line 286 of
lib/lp/code/model/revision.py RevisionSet.new was the primary culprit,
generating most of the SELECTS and INSERTS.

That's all I have for the moment.

-- 
Benji York