launchpad-dev team mailing list archive

[Jeroen Vermeulen <jtv@xxxxxxxxxxxxx>]

Hi folks,

Benji and I got a little worked up yesterday over the time it takes to 
do a "make" or "make schema."  We're probably not alone.

I have two pieces of good news for you, then some pie in the sky, and a 
question (with more floating pie attached).


Good news #1: an optimized version of security.py just hit devel, so 
"make schema" will be much faster.


Good news #2: the default "make" target now supports parallel builds!

So instead of "make" or "make build" try "make -j2" or "make build -j2" 
respectively.  On my system that speeds it up from 2:40 to 1:30, give or 
take.  (Parallel builds may still break on some other targets though.)


Pie in the sky #1: Benji figured a way around a sizable chunk of the 
time it takes to generate the WADL.  WADL generation is the big 
bottleneck, especially in a parallel build.


Then, my Question: Would anyone mind if we stopped doing a full build 
for every schema build?

It would mean that you can't do only a "make schema" in your branch and 
expect LP or its tests to run; you'll have to do a "make" to set up the 
necessary files in your branch, and optionally a "make schema" to set up 
the database according to the branch you're in.


And Pie in the sky #2: With that change in the makefile, I can do a 
"make schema" in under 50 seconds.  Even in a branch that hasn't been 
built yet.


Happy Hacking!  Details below for whoever wants them.


Jeroen


Appendix A: Technical Details

 * We made security.py faster, but not yet incremental like we want to. 
 The only change was that a lot of GRANT statements were bundled 
together into fewer, bigger statements.  We could extend that same trick 
to get further speedups, but ZCML processing is now the bottleneck.

 * On my machine, "make -j2" manages to keep 2 cores productive for 
almost the entire build.  But "make -j3" merely slows down the longest 
job while pointlessly finishing the shorter ones faster.

 * WADL generation now dominates build time, so for further gains we 
need that to run either faster or in parallel.  Breaking it down into 
multiple script runs would incur more ZCML processing, which could be 
significant.

 * The cost sink in WADL generation is almost half a million calls to 
add_option in optparse.  Accounted for about 1 minute in a 4-minute 
profiling run.  The module was never meant for such heavy use, but Benji 
thinks we could probably get by without those calls.

 * Avoiding the full build during "make schema" can be done by 
replacing this Makefile line:

schema: build clean_codehosting

with this one:

schema: buildout_bin clean_codehosting

EOF