launchpad-reviewers team mailing list archive

[William Grant <me@xxxxxxxxxxxxxxxxxx>]

William Grant has proposed merging lp:~wgrant/launchpad/nightly.sh-fixes into lp:launchpad.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)
Related bugs:
  #712282 daily.sh is opaque to developers
  https://bugs.launchpad.net/bugs/712282
  #712285 update-pkgcache.py should not be run in nightly.sh
  https://bugs.launchpad.net/bugs/712285

For more details, see:
https://code.launchpad.net/~wgrant/launchpad/nightly.sh-fixes/+merge/48441

nightly.sh is currently pretty opaque, sending timestamps of each step to launchpad-error-reports, and that's about it. This branch adjusts it to be more transparent, logging each of the scripts (with DEBUG, except where that would produce too much output) to a separate file, and logging its own timestamps to nightly.log. All these files will live in a directory that was recently created on loganberry.

I've also removed update-pkgcache.py, since it takes 15 hours. This should get nightly.sh down to around 17 hours. It will be added directly into a crontab after this is rolled out.
-- 
https://code.launchpad.net/~wgrant/launchpad/nightly.sh-fixes/+merge/48441
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~wgrant/launchpad/nightly.sh-fixes into lp:launchpad.

=== modified file 'cronscripts/nightly.sh'
--- cronscripts/nightly.sh	2011-01-19 00:03:57 +0000
+++ cronscripts/nightly.sh	2011-02-03 10:05:00 +0000
@@ -31,53 +31,59 @@
 export http_proxy=http://squid.internal:3128/
 export ftp_proxy=http://squid.internal:3128/
 
+LOGDIR=/srv/launchpad.net/production-logs/nightly
+LOGFILE=$LOGDIR/nightly.log
+
 LOCK=/var/lock/launchpad_nightly.lock
 lockfile -r0 -l 259200 $LOCK
 if [ $? -ne 0 ]; then
-    echo Unable to grab $LOCK lock - aborting
+    echo `date`: Unable to grab $LOCK lock - aborting | tee -a $LOGFILE
     ps fuxwww
     exit 1
 fi
 
 cd /srv/launchpad.net/production/launchpad/cronscripts
 
-echo == Expiring memberships `date` ==
-python -S flag-expired-memberships.py -q
-
-echo == Allocating revision karma `date` ==
-python -S allocate-revision-karma.py -q
-
-echo == Recalculating karma `date` ==
-python -S foaf-update-karma-cache.py -q
-
-echo == Updating cached statistics `date` ==
-python -S update-stats.py -q
-
-echo == Expiring questions `date` ==
-python -S expire-questions.py
+echo `date`: Grabbed lock >> $LOGFILE
+
+echo `date`: Expiring memberships >> $LOGFILE
+python -S flag-expired-memberships.py -q --log-file=DEBUG:$LOGDIR/flag-expired-memberships.log
+
+echo `date`: Allocating revision karma >> $LOGFILE
+python -S allocate-revision-karma.py -q --log-file=DEBUG:$LOGDIR/allocate-revision-karma.log
+
+echo `date`: Recalculating karma >> $LOGFILE
+python -S foaf-update-karma-cache.py -q --log-file=INFO:$LOGDIR/foaf-update-karma-cache.log
+
+echo `date`: Updating cached statistics >> $LOGFILE
+python -S update-stats.py -q --log-file=DEBUG:$LOGDIR/update-stats.log
+
+echo `date`: Expiring questions >> $LOGFILE
+python -S expire-questions.py -q --log-file=DEBUG:$LOGDIR/expire-questions.log
 
 ### echo == Expiring bugs `date` ==
 ### python -S expire-bugtasks.py
 
-# checkwatches.py is scheduled in ~launchpad-pqn/lp-production-crontabs.
+# checkwatches.py is scheduled in lp-production-crontabs.
 ### echo == Updating bug watches `date` ==
 ### python -S checkwatches.py
 
-echo == Updating bugtask target name caches `date` ==
-python -S update-bugtask-targetnamecaches.py -q
-
-echo == Updating personal standings `date` ==
-python -S update-standing.py -q
-
-echo == Updating CVE database `date` ==
-python -S update-cve.py -q
-
-echo == Updating package cache `date` ==
-python -S update-pkgcache.py -q
-
-# Release finder is scheduled in ~launchpad-pqn/lp-production-crontabs.
+echo `date`: Updating bugtask target name caches >> $LOGFILE
+python -S update-bugtask-targetnamecaches.py -q --log-file=DEBUG:$LOGDIR/update-bugtask-targetnamecaches.log
+
+echo `date`: Updating personal standings >> $LOGFILE
+python -S update-standing.py -q --log-file=DEBUG:$LOGDIR/update-standing.log
+
+echo `date`: Updating CVE database >> $LOGFILE
+python -S update-cve.py -q --log-file=DEBUG:$LOGDIR/update-cve.log
+
+# update-pkgcache.py is scheduled in lp-production-crontabs.
+#echo == Updating package cache `date` ==
+#python -S update-pkgcache.py -q
+
+# Release finder is scheduled in lp-production-crontabs.
 #echo == Product Release Finder `date` ==
 #python -S product-release-finder.py -q
 
-
+echo `date`: Removing lock >> $LOGFILE
 rm -f $LOCK