launchpad-reviewers team mailing list archive

[Colin Watson <mp+436426@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad-buildd:adjust-oom-score into launchpad-buildd:master.

Commit message:
Disable OOM killing of launchpad-buildd

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad-buildd/+git/launchpad-buildd/+merge/436426

This came up as an idea at the last engineering sprint.  If a builder is under memory pressure, we think that the kernel will sometimes choose to kill the launchpad-buildd process, which is the worst thing it could choose to do: the machine might as well be dead at that point.  We'd always rather that it kill some other process so that we have a better chance of getting a useful build log.

See systemd.exec(5).
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad-buildd:adjust-oom-score into launchpad-buildd:master.

diff --git a/debian/changelog b/debian/changelog
index 87e8c12..3bc7d49 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+launchpad-buildd (229) UNRELEASED; urgency=medium
+
+  * Prevent the kernel from killing launchpad-buildd if it runs out of
+    memory.
+
+ -- Colin Watson <cjwatson@xxxxxxxxxx>  Fri, 27 Jan 2023 10:38:34 +0000
+
 launchpad-buildd (228) focal; urgency=medium
 
   * In CI jobs, chown the VCS tree to buildd:buildd after fetching it, since
diff --git a/debian/launchpad-buildd@.service b/debian/launchpad-buildd@.service
index 9d4b4b2..6ec785c 100644
--- a/debian/launchpad-buildd@.service
+++ b/debian/launchpad-buildd@.service
@@ -26,6 +26,9 @@ Environment=BUILDD_CONFIG=/etc/launchpad-buildd/%i
 ExecStartPre=/usr/bin/test ${RUN_NETWORK_REQUESTS_AS_ROOT} = yes
 ExecStartPre=/usr/bin/install -d /home/buildd/filecache-default
 ExecStart=/usr/bin/twistd3 --no_save --pidfile /run/launchpad-buildd/%i.pid --python /usr/lib/launchpad-buildd/buildd.tac --logfile /var/log/launchpad-buildd/%i.log --umask 022 --nodaemon
+# If the machine runs out of memory, killing just about any other process is
+# better than killing launchpad-buildd.
+OOMScoreAdjust=-1000
 
 [Install]
 WantedBy=multi-user.target