launchpad-reviewers team mailing list archive

[Stuart Bishop <stuart.bishop@xxxxxxxxxxxxx>]

Stuart Bishop has proposed merging lp:~stub/launchpad/trivial into lp:launchpad.

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)
Related bugs:
  Bug #951401 in Launchpad itself: "parse-ppa-apache-logs failing (missing files)"
  https://bugs.launchpad.net/launchpad/+bug/951401
  Bug #1263002 in Launchpad itself: "Twisted feature flag support fails typecasting when updating"
  https://bugs.launchpad.net/launchpad/+bug/1263002

For more details, see:
https://code.launchpad.net/~stub/launchpad/trivial/+merge/201335

Stop staging restores from filling available disk with unshipped replication data.

To apply the brakes, we add a script to the database restore pipeline that blocks the pipe when there is too much unshipped data hanging around.
-- 
https://code.launchpad.net/~stub/launchpad/trivial/+merge/201335
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~stub/launchpad/trivial into lp:launchpad.

=== modified file 'database/replication/Makefile'
--- database/replication/Makefile	2013-09-23 04:29:31 +0000
+++ database/replication/Makefile	2014-01-13 06:35:23 +0000
@@ -70,8 +70,10 @@
 	# production to users not maintained by security.py.
 	pg_restore --list ${STAGING_DUMP} | grep -v 'TRIGGER public _sl_' \
 	    > ${DUMPLIST}
-	pg_restore --dbname=lpmain_staging --no-owner ${EXIT_ON_ERROR} \
-	    --use-list=${DUMPLIST} ${MULTIPROC} ${STAGING_DUMP}
+	cat ${STAGING_DUMP} \
+	    | ./walblock.py -d /var/lib/postgresql/9.1/staging/pg_xlog \
+	    | pg_restore --dbname=lpmain_staging --no-owner ${EXIT_ON_ERROR} \
+		--use-list=${DUMPLIST} ${MULTIPROC}
 	rm ${DUMPLIST}
 	# Apply database patches.
 	@echo Running upgrade.py `date`.

=== added file 'database/replication/walblock.py'
--- database/replication/walblock.py	1970-01-01 00:00:00 +0000
+++ database/replication/walblock.py	2014-01-13 06:35:23 +0000
@@ -0,0 +1,58 @@
+#!/usr/bin/python
+#
+# Copyright 2014 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Feed stdin to stout, blocking if there are too many unshipped WAL files"""
+
+__metaclass__ = type
+__all__ = []
+
+from glob import glob
+from optparse import OptionParser
+import os.path
+import sys
+import time
+
+
+def main():
+    parser = OptionParser()
+    parser.add_option(
+        "-n", dest="num_ready", metavar="N", type="int",
+        help="Block if there are more than N unshipped WAL files.", default=25)
+    parser.add_option(
+        "-d", dest="wal_dir", metavar="DIR", type="string",
+        help="Path to pg_xlog directory",
+        default="/var/lib/postgresql/9.1/main/pg_xlog")
+    parser.add_option(
+        "-v", "--verbose", action="store_true", default=False, help="Verbose")
+    #logger_options(parser)
+    #db_options(parser)
+    options, args = parser.parse_args()
+    if args:
+        parser.error('Too many arguments')
+
+    chunk_size = 1024*1024
+
+    ready_wal_glob = os.path.join(options.wal_dir, 'archive_status', '*.ready')
+
+    while True:
+        notified = False
+        while len(glob(ready_wal_glob)) > options.num_ready:
+            if options.verbose and not notified:
+                notified = True
+                print >> sys.stderr, 'Blocking on {0} unshipped WAL'.format(
+                    len(glob(ready_wal_glob))),
+            time.sleep(5)
+        if options.verbose and notified:
+            print >> sys.stderr, '... Done'
+
+        chunk = sys.stdin.read(chunk_size)
+        if chunk == '':
+            sys.stdout.flush()
+            return 0
+        sys.stdout.write(chunk)
+
+
+if __name__ == '__main__':
+    raise SystemExit(main())