launchpad-reviewers team mailing list archive

["Jeroen T. Vermeulen" <jtv@xxxxxxxxxxxxx>]

Jeroen T. Vermeulen has proposed merging lp:~jtv/launchpad/parallel-make into lp:launchpad/devel.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers): code


= Parallel make =

This fixes the Makefile so that the default target (or "build") can be built in parallel.  So instead of "make" you can do "make -j2" and get a nice speedup: on my system right after a "make clean" it takes the build time from 2:43 to 1:32.  Should be a welcome change.

What changes?  A single buildout run in the makefile produces bin/py and a bunch of other files.  But "make" isn't aware that there's no point to running two buildout instances at once, and indeed it's likely to break.

So I channeled all the files produced by buildout through a single virtual target.  Now "make" realizes when it's about to run two buildout instances in parallel that they are actually producing the same target—and wait for the existing run to complete instead of starting a redundant one.

(Actually I had to start out with completing dependencies on bin/py, because not all targets that used it depended on it.  These things become more noticeable when you run parallel builds.)

The good news is that the build parallelizes almost perfectly to two processes; the bad news is that that's where the ride ends.  This is because one single-process job, the generation of WADL files, takes up just about half the build time.  With two processors, the two halvse will complete almost simultaneously at just over half the single-process build time.  Adding a third CPU won't help the WADL target, and so can't do much for overall build time.  At best it gets you to the point where everything waits for that one big job a little faster.  On my dual-core SMT laptop, a third process even slows the build down a little.


Jeroen
-- 
https://code.launchpad.net/~jtv/launchpad/parallel-make/+merge/41108
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~jtv/launchpad/parallel-make into lp:launchpad/devel.

=== modified file 'Makefile'
--- Makefile	2010-11-02 01:34:05 +0000
+++ Makefile	2010-11-17 22:07:46 +0000
@@ -55,10 +55,10 @@
 newsampledata:
 	$(MAKE) -C database/schema newsampledata
 
-hosted_branches: $(PY)
+hosted_branches: buildout_bin
 	$(PY) ./utilities/make-dummy-hosted-branches
 
-$(API_INDEX): $(BZR_VERSION_INFO)
+$(API_INDEX): $(BZR_VERSION_INFO) buildout_bin
 	mkdir -p $(APIDOC_DIR).tmp
 	LPCONFIG=$(LPCONFIG) $(PY) ./utilities/create-lp-wadl-and-apidoc.py --force "$(WADL_TEMPLATE)"
 	mv $(APIDOC_DIR).tmp $(APIDOC_DIR)
@@ -66,12 +66,12 @@
 apidoc: compile $(API_INDEX)
 
 # Run by PQM.
-check_merge: $(PY)
+check_merge: buildout_bin
 	[ `PYTHONPATH= bzr status -S database/schema/ | \
 		grep -v "\(^P\|pending\|security.cfg\|Makefile\|unautovacuumable\|_pythonpath.py\)" | wc -l` -eq 0 ]
 	${PY} lib/canonical/tests/test_no_conflict_marker.py
 
-check_db_merge: $(PY)
+check_db_merge: buildout_bin
 	${PY} lib/canonical/tests/test_no_conflict_marker.py
 
 check_config: build
@@ -109,16 +109,16 @@
 	${PY} -t ./test_on_merge.py $(VERBOSITY) $(TESTOPTS) \
 		--layer=MailmanLayer
 
-lint: ${PY}
+lint: buildout_bin
 	@bash ./bin/lint.sh
 
-lint-verbose: ${PY}
+lint-verbose: buildout_bin
 	@bash ./bin/lint.sh -v
 
-xxxreport: $(PY)
+xxxreport: buildout_bin
 	${PY} -t ./utilities/xxxreport.py -f csv -o xxx-report.csv ./
 
-check-configs: $(PY)
+check-configs: buildout_bin
 	${PY} utilities/check-configs.py
 
 pagetests: build
@@ -145,7 +145,7 @@
 	# roll-up files.  They fiddle with built-in functions!  See Bug 482340.
 	${SHHH} bin/jsbuild $(JSFLAGS) -b $(LAZR_BUILT_JS_ROOT) -x testing/ -c $(LAZR_BUILT_JS_ROOT)/yui
 
-jsbuild: jsbuild_lazr bin/jsbuild bin/jssize
+jsbuild: jsbuild_lazr bin/jsbuild bin/jssize buildout_bin
 	${SHHH} bin/jsbuild \
 		$(JSFLAGS) \
 		-n launchpad \
@@ -173,7 +173,7 @@
 	@exit 1
 endif
 
-buildonce_eggs: $(PY)
+buildonce_eggs: buildout_bin
 	find eggs -name '*.pyc' -exec rm {} \;
 
 # The download-cache dependency comes *before* eggs so that developers get the
@@ -192,19 +192,21 @@
 # and the other bits might run into problems like bug 575037.  This
 # target runs buildout, and then removes everything created except for
 # the eggs.
-build_eggs: $(BUILDOUT_BIN) clean_buildout
+build_eggs: buildout_bin clean_buildout
+
+$(BUILDOUT_BIN): buildout_bin
 
 # This builds bin/py and all the other bin files except bin/buildout.
 # Remove the target before calling buildout to ensure that buildout
 # updates the timestamp.
-$(BUILDOUT_BIN): bin/buildout versions.cfg $(BUILDOUT_CFG) setup.py
+buildout_bin: bin/buildout versions.cfg $(BUILDOUT_CFG) setup.py
 	$(RM) $@
 	$(SHHH) PYTHONPATH= ./bin/buildout \
                 configuration:instance_name=${LPCONFIG} -c $(BUILDOUT_CFG)
 
 # bin/compile_templates is responsible for building all chameleon templates,
 # of which there is currently one, but of which many more are coming.
-compile: $(PY) $(BZR_VERSION_INFO)
+compile: buildout_bin $(BZR_VERSION_INFO)
 	mkdir -p /var/tmp/vostok-archive
 	${SHHH} $(MAKE) -C sourcecode build PYTHON=${PYTHON} \
 	    LPCONFIG=${LPCONFIG}
@@ -463,7 +465,7 @@
 		--docformat restructuredtext --verbose-about epytext-summary \
 		$(PYDOCTOR_OPTIONS)
 
-.PHONY: apidoc check tags TAGS zcmldocs realclean clean debug stop\
+.PHONY: apidoc buildout_bin check tags TAGS zcmldocs realclean clean debug stop\
 	start run ftest_build ftest_inplace test_build test_inplace pagetests\
 	check check_merge \
 	schema default launchpad.pot check_merge_ui pull scan sync_branches\