launchpad-reviewers team mailing list archive

[Gavin Panella <gavin.panella@xxxxxxxxxxxxx>]

Gavin Panella has proposed merging lp:~allenap/launchpad/link-sourcecode-faster into lp:launchpad.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~allenap/launchpad/link-sourcecode-faster/+merge/63081

Aaron recently sped up the fetching of sourcecode in rocketfuel-get, and now I notice that linking external sourcecode for all local branches takes a long time (especially for something that's fairly simple). This branch changes rocketfuel-get to run link-external-sourcecode in parallel, with up to $cpucount processes. On my machine - which reports 4 cores - this speeds things up by about 70% (from ~9s to <3s).
-- 
https://code.launchpad.net/~allenap/launchpad/link-sourcecode-faster/+merge/63081
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~allenap/launchpad/link-sourcecode-faster into lp:launchpad.

=== modified file 'utilities/rocketfuel-get'
--- utilities/rocketfuel-get	2011-05-29 06:33:23 +0000
+++ utilities/rocketfuel-get	2011-06-01 09:07:47 +0000
@@ -10,6 +10,9 @@
 # Stop if there's an error, and treat unset variables as errors.
 set -eu
 
+# A rough measure of how much stuff we can do in parallel.
+CPU_COUNT="$(egrep -c '^processor\b' /proc/cpuinfo)"
+
 # Helper function to run a child process, indenting stdout to aid
 # readability.
 run-child() {
@@ -56,23 +59,18 @@
 fi
 
 echo "Updating sourcecode dependencies in rocketfuel:"
-echo ""
-echo "    Sourcedeps: $LP_SOURCEDEPS_PATH"
-
-run-child $LP_TRUNK_PATH/utilities/update-sourcecode $LP_SOURCEDEPS_PATH $sourcedeps_conf
-
+run-child \
+    "$LP_TRUNK_PATH/utilities/update-sourcecode" \
+    "$LP_SOURCEDEPS_PATH" "$sourcedeps_conf"
 
 # Update the current trees in the repo.
 echo "Updating sourcecode dependencies in current local branches:"
-for branch in "$LP_PROJECT_PATH"/*
-do
-    if [ -d "$branch/sourcecode" ]
-    then
-        echo "    ${branch}"
-        run-child "$LP_TRUNK_PATH/utilities/link-external-sourcecode" \
-            --parent="$LP_PROJECT_ROOT/$LP_SOURCEDEPS_DIR" --target="$branch"
-    fi
-done
+find "$LP_PROJECT_PATH" -mindepth 2 -maxdepth 2 \
+    -type d -name sourcecode -printf '%h\0' | \
+    run-child xargs --no-run-if-empty \
+        --max-procs="${CPU_COUNT}" --max-args=1 --null \
+        "$LP_TRUNK_PATH/utilities/link-external-sourcecode" \
+            --parent "$LP_PROJECT_ROOT/$LP_SOURCEDEPS_DIR" --target
 
 # Build launchpad if there were changes.
 if [ $FINAL_REV != $INITIAL_REV ];