launchpad-reviewers team mailing list archive

[Colin Watson <mp+413048@xxxxxxxxxxxxxxxxxx>]

Colin Watson has proposed merging ~cjwatson/launchpad:buildd-manager-file-limit into launchpad:master.

Commit message:
Update buildd-manager's open file descriptor limit

Requested reviews:
  Launchpad code reviewers (launchpad-reviewers)

For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/413048

We added 30 new builders to Launchpad's production build farm today (for a total of 348), after which we started seeing "Too many open files" from buildd-manager.  This is because the value we set for `RLIMIT_NOFILE` was tuned for the old in-process download strategy, which we replaced in July 2020.  Re-tune it for the new process-pool-based download strategy.

I also updated the description and default for `config.builddmaster.download_connections`, which were similarly based on the old in-process download strategy.  The value on production was already set to 128.
-- 
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:buildd-manager-file-limit into launchpad:master.

diff --git a/daemons/buildd-manager.tac b/daemons/buildd-manager.tac
index 2612d9a..015b877 100644
--- a/daemons/buildd-manager.tac
+++ b/daemons/buildd-manager.tac
@@ -27,9 +27,12 @@ dbconfig.override(dbuser='buildd_manager', isolation_level='read_committed')
 # Should be removed from callsites verified to not need it.
 set_immediate_mail_delivery(True)
 
-# Allow generous slack for database connections, idle download connections,
-# etc.
-soft_nofile = config.builddmaster.download_connections + 1024
+# ampoule uses five file descriptors per subprocess (i.e.
+# 5 * config.builddmaster.download_connections); we also need at least three
+# per active builder for resuming virtualized builders or making XML-RPC
+# calls, and we also need to allow slack for odds and ends like database
+# connections.
+soft_nofile = 5 * config.builddmaster.download_connections + 2048
 _, hard_nofile = resource.getrlimit(resource.RLIMIT_NOFILE)
 resource.setrlimit(resource.RLIMIT_NOFILE, (soft_nofile, hard_nofile))
 
diff --git a/lib/lp/services/config/schema-lazr.conf b/lib/lp/services/config/schema-lazr.conf
index 73bfb70..ef17be9 100644
--- a/lib/lp/services/config/schema-lazr.conf
+++ b/lib/lp/services/config/schema-lazr.conf
@@ -67,9 +67,9 @@ socket_timeout: 40
 # datatype: integer
 virtualized_socket_timeout: 30
 
-# The maximum number of file download connections that may be open
-# across all builders.
-download_connections: 2048
+# The maximum number of child processes to run for downloading files from
+# builders.
+download_connections: 128
 
 # Activate the Build Notification system.
 # datatype: boolean