launchpad-reviewers team mailing list archive

[Colin Watson <mp+439236@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:debbugs-sync into launchpad:master.

Commit message:
Add debbugs-sync script

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/439236

This adopts an ad-hoc script currently running on a Canonical machine to sync data from the Debian bug tracking system for use by Launchpad bug watches.

There's definitely scope for cleaning this up further, adding some kind of tests, and so on.  For now, I'm working on the theory that it's better to get production code into more visible and maintainable places even if it's not in an ideal state, so I've only done some minor cleanup to pacify `shellcheck` and tidy up logging slightly, as well as excluding some large files that we don't need in order to reduce load on both ends of the `rsync`.

See also https://portal.admin.canonical.com/C136072.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:debbugs-sync into launchpad:master.

diff --git a/scripts/debbugs-sync b/scripts/debbugs-sync
new file mode 100755
index 0000000..66be6bb
--- /dev/null
+++ b/scripts/debbugs-sync
@@ -0,0 +1,49 @@
+#! /bin/sh
+#
+# Copyright 2023 Canonical Ltd.  This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+set -e
+
+TO=/srv/bugs-mirror.debian.org
+FROM=bugs-mirror.debian.org
+
+umask 002
+mkdir -p "$TO"
+
+HOSTNAME=$(hostname -f)
+LOCK="$TO/Mirror-Update-in-Progress-$HOSTNAME"
+
+# Check to see if another sync is in progress
+if lockfile -! -l 43200 -r 0 "$LOCK"; then
+    echo "$HOSTNAME is unable to start rsync, lock file exists" >&2
+    exit 1
+fi
+cleanup () {
+    rm -f "$LOCK"
+}
+trap cleanup EXIT
+
+cd "$TO"
+
+# Current bugs
+rsync -rtlHv --delete --stats \
+      --exclude "Mirror-Update-in-Progress-$HOSTNAME" \
+      --exclude "-*.log" \
+      $FROM::bts-spool-db/ db-h/
+
+# Index
+rsync -rtlHv --delete --stats \
+      --exclude "Mirror-Update-in-Progress-$HOSTNAME" \
+      --exclude '*.idx' --exclude '*.idx-new' \
+      $FROM::bts-spool-index/ index/
+
+# Archived bugs
+rsync -rtlHv --delete --stats \
+      --exclude "Mirror-Update-in-Progress-$HOSTNAME" \
+      $FROM::bts-spool-archive/ archive/
+
+mkdir -p "$TO/trace"
+date -u > "$TO/trace/$HOSTNAME"
+
+rm -f "$LOCK"