launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #29144
[Merge] ~cjwatson/launchpad:charm-appserver into launchpad:master
Colin Watson has proposed merging ~cjwatson/launchpad:charm-appserver into launchpad:master.
Commit message:
Add initial appserver charm
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~cjwatson/launchpad/+git/launchpad/+merge/429558
There are still a lot more details of configuration to flesh out based on lp:lp-production-configs, and some more things to set up such as Nagios checks, an MTA, and cron jobs; but this gets us to the point where it's possible to run a very minimal appserver and make local requests to it, provided that you can separately arrange for a database that it can connect to.
I've opted for the approach of generating `lazr.config` files based on charm configuration. This wasn't a completely obvious decision, since it potentially means copying quite a lot of Launchpad's config schema into charm `config.yaml` files. However, as long as we're careful about how we go about this (looking for patterns so that we can have a smaller set of charm configuration keys), it has the benefit that it should be much easier to avoid drift between our multiple environments.
--
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/launchpad:charm-appserver into launchpad:master.
diff --git a/charm/Makefile b/charm/Makefile
index 5bb96fe..bad8b8a 100644
--- a/charm/Makefile
+++ b/charm/Makefile
@@ -16,7 +16,7 @@ BUILD_LABEL = $(shell git rev-parse HEAD)
TARBALL = $(APP_NAME).tar.gz
ASSET = ../build/$(BUILD_LABEL)/$(TARBALL)
-CHARMS := launchpad
+CHARMS := launchpad-appserver
all: ## alias to build
all: build
@@ -44,6 +44,9 @@ build: $(foreach charm,$(CHARMS),build-$(charm))
build-launchpad: ## build the launchpad charm
build-launchpad: dist/$(call charm_file,launchpad)
+build-launchpad-appserver: ## build the launchpad-appserver charm
+build-launchpad-appserver: dist/$(call charm_file,launchpad-appserver)
+
dist/%_ubuntu-$(CHARM_SERIES)-$(ARCH).charm: $(CHARM_DEPS) | $(BUILDDIR)
echo "Building $*..."
rm -rf $*/tmp
diff --git a/charm/bundle.yaml.in b/charm/bundle.yaml.in
index baecf0a..e5c5c67 100644
--- a/charm/bundle.yaml.in
+++ b/charm/bundle.yaml.in
@@ -1,8 +1,8 @@
series: bionic
description: "Launchpad development bundle"
applications:
- launchpad:
- charm: ./dist/launchpad_ubuntu-18.04-amd64.charm
+ launchpad-appserver:
+ charm: ./dist/launchpad-appserver_ubuntu-18.04-amd64.charm
num_units: 1
options:
build_label: "%BUILD_LABEL%"
diff --git a/charm/launchpad-appserver/charmcraft.yaml b/charm/launchpad-appserver/charmcraft.yaml
new file mode 100644
index 0000000..a6f2757
--- /dev/null
+++ b/charm/launchpad-appserver/charmcraft.yaml
@@ -0,0 +1,20 @@
+type: charm
+bases:
+ - build-on:
+ - name: ubuntu
+ channel: "18.04"
+ run-on:
+ - name: ubuntu
+ channel: "18.04"
+parts:
+ launchpad-appserver:
+ source: .
+ plugin: reactive
+ build-snaps: [charm]
+ build-environment:
+ - CHARM_LAYERS_DIR: tmp/deps/ols-layers/layer
+ - CHARM_INTERFACES_DIR: tmp/deps/ols-layers/interface
+ - PIP_NO_INDEX: "true"
+ - PIP_FIND_LINKS: tmp/deps/charm-wheels
+ stage:
+ - -tmp
diff --git a/charm/launchpad-appserver/config.yaml b/charm/launchpad-appserver/config.yaml
new file mode 100644
index 0000000..8967f67
--- /dev/null
+++ b/charm/launchpad-appserver/config.yaml
@@ -0,0 +1,35 @@
+options:
+ dbuser:
+ type: string
+ description: The database user used with the main database.
+ default: ""
+ dbuser_session:
+ type: string
+ description: The database user used with the session database.
+ default: ""
+ devmode:
+ type: boolean
+ description: Is this server running in dev mode?
+ default: true
+ min_legitimate_account_age:
+ type: int
+ description: Minimum account age in days that indicates a legitimate user.
+ default: 0
+ min_legitimate_karma:
+ type: int
+ description: Minimum karma value that indicates a legitimate user.
+ default: 0
+ port_main:
+ type: int
+ description: Port for the main application server.
+ default: 8085
+ port_xmlrpc:
+ type: int
+ description: Port for the XML-RPC application server.
+ default: 8087
+ wsgi_workers:
+ type: int
+ default: 0
+ description: >
+ The number of worker processes for handling requests.
+ The default is 0, indicating twice the number of CPUs plus one.
diff --git a/charm/launchpad-appserver/layer.yaml b/charm/launchpad-appserver/layer.yaml
new file mode 100644
index 0000000..bcff867
--- /dev/null
+++ b/charm/launchpad-appserver/layer.yaml
@@ -0,0 +1,3 @@
+includes:
+ - layer:launchpad-base
+repo: https://git.launchpad.net/launchpad
diff --git a/charm/launchpad-appserver/metadata.yaml b/charm/launchpad-appserver/metadata.yaml
new file mode 100644
index 0000000..321f110
--- /dev/null
+++ b/charm/launchpad-appserver/metadata.yaml
@@ -0,0 +1,15 @@
+name: launchpad-appserver
+display-name: launchpad-appserver
+summary: Launchpad application server
+maintainer: Colin Watson <cjwatson@xxxxxxxxxxxxx>
+description: |
+ Launchpad is an open source suite of tools that help people and teams
+ to work together on software projects.
+
+ This charm runs a Launchpad application server.
+tags:
+ # https://juju.is/docs/charm-metadata#heading--charm-store-fields
+ - network
+series:
+ - bionic
+subordinate: false
diff --git a/charm/launchpad-appserver/reactive/launchpad-appserver.py b/charm/launchpad-appserver/reactive/launchpad-appserver.py
new file mode 100644
index 0000000..c5fac0e
--- /dev/null
+++ b/charm/launchpad-appserver/reactive/launchpad-appserver.py
@@ -0,0 +1,108 @@
+# Copyright 2022 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import subprocess
+from multiprocessing import cpu_count
+
+from charmhelpers.core import hookenv, host, templating
+from charms.launchpad.base import (
+ config_file_path,
+ configure_lazr,
+ get_service_config,
+ lazr_config_files,
+)
+from charms.reactive import helpers, set_state, when, when_not
+from ols import base
+
+
+def reload_or_restart(service):
+ subprocess.run(["systemctl", "reload-or-restart", service], check=True)
+
+
+def enable_service(service):
+ subprocess.run(["systemctl", "enable", service], check=True)
+
+
+@host.restart_on_change(
+ {
+ "/etc/rsyslog.d/22-launchpad.conf": ["rsyslog"],
+ "/lib/systemd/system/launchpad.service": ["launchpad"],
+ config_file_path("gunicorn.conf.py"): ["launchpad"],
+ },
+ restart_functions={
+ "rsyslog": reload_or_restart,
+ "gunicorn": enable_service,
+ },
+)
+def configure_gunicorn(config):
+ hookenv.log("Writing gunicorn configuration.")
+ config = dict(config)
+ if config["wsgi_workers"] == 0:
+ config["wsgi_workers"] = cpu_count() * 2 + 1
+ templating.render(
+ "gunicorn.conf.py.j2", config_file_path("gunicorn.conf.j2"), config
+ )
+ templating.render(
+ "launchpad.service.j2", "/lib/systemd/system/launchpad.service", config
+ )
+ host.add_user_to_group("syslog", base.user())
+ templating.render("rsyslog.j2", "/etc/rsyslog.d/22-launchpad.conf", config)
+
+
+def configure_logrotate(config):
+ hookenv.log("Writing logrotate configuration.")
+ templating.render(
+ "logrotate.conf.j2",
+ "/etc/logrotate.d/launchpad",
+ config,
+ perms=0o644,
+ )
+
+
+def restart(soft=False):
+ if soft:
+ reload_or_restart("launchpad")
+ else:
+ host.service_restart("launchpad")
+
+
+def config_files():
+ files = []
+ files.extend(lazr_config_files())
+ files.append(config_file_path("launchpad-appserver/launchpad-lazr.conf"))
+ return files
+
+
+@when("launchpad.base.configured")
+@when_not("service.configured")
+def configure():
+ config = get_service_config()
+ # XXX cjwatson 2022-09-07: Some config items have no reasonable default.
+ # We should set the workload status to blocked in that case.
+ configure_lazr(
+ config,
+ "launchpad-appserver-lazr.conf",
+ "launchpad-appserver/launchpad-lazr.conf",
+ )
+ configure_gunicorn(config)
+ configure_logrotate(config)
+
+ restart_type = None
+ if helpers.any_file_changed(
+ [base.version_info_path(), "/lib/systemd/system/launchpad.service"]
+ ):
+ restart_type = "hard"
+ elif helpers.any_file_changed(config_files()):
+ restart_type = "soft"
+ if restart_type is None:
+ hookenv.log("Not restarting, since no config files were changed")
+ else:
+ hookenv.log(f"Config files changed; performing {restart_type} restart")
+ restart(soft=(restart_type == "soft"))
+
+ set_state("service.configured")
+
+
+@when("service.configured")
+def check_is_running():
+ hookenv.status_set("active", "Ready")
diff --git a/charm/launchpad-appserver/templates/gunicorn.conf.py.j2 b/charm/launchpad-appserver/templates/gunicorn.conf.py.j2
new file mode 100644
index 0000000..efd8a1e
--- /dev/null
+++ b/charm/launchpad-appserver/templates/gunicorn.conf.py.j2
@@ -0,0 +1,9 @@
+bind = [":{{ port_main }}", ":{{ port_xmlrpc }}"]
+workers = {{ wsgi_workers }}
+# This is set relatively low to work around memory leaks on Python 3.
+max_requests = 2000
+log_level = "DEBUG"
+# Must be higher than the highest hard_timeout feature rule.
+timeout = 65
+graceful_timeout = 120
+
diff --git a/charm/launchpad-appserver/templates/launchpad-appserver-lazr.conf b/charm/launchpad-appserver/templates/launchpad-appserver-lazr.conf
new file mode 100644
index 0000000..dfa56bd
--- /dev/null
+++ b/charm/launchpad-appserver/templates/launchpad-appserver-lazr.conf
@@ -0,0 +1,23 @@
+# Public configuration data. The contents of this file may be freely shared
+# with developers if needed for debugging.
+
+# A schema's sections, keys, and values are automatically inherited, except
+# for '.optional' sections. Update this config to override key values.
+# Values are strings, except for numbers that look like ints. The tokens
+# true, false, and none are treated as True, False, and None.
+
+[meta]
+extends: ../launchpad-base-lazr.conf
+
+[launchpad]
+{%- if dbuser %}
+dbuser: {{ dbuser }}
+{%- endif %}
+devmode: {{ devmode }}
+min_legitimate_account_age: {{ min_legitimate_account_age }}
+min_legitimate_karma: {{ min_legitimate_karma }}
+
+[launchpad_session]
+{%- if dbuser_session %}
+dbuser: {{ dbuser_session }}
+{%- endif %}
diff --git a/charm/launchpad-appserver/templates/launchpad.service.j2 b/charm/launchpad-appserver/templates/launchpad.service.j2
new file mode 100644
index 0000000..fb5c372
--- /dev/null
+++ b/charm/launchpad-appserver/templates/launchpad.service.j2
@@ -0,0 +1,24 @@
+[Unit]
+Description=Launchpad application server
+After=network.target
+ConditionPathExists=!{{ code_dir }}/maintenance.txt
+
+[Service]
+User=launchpad
+Group=launchpad
+WorkingDirectory={{ code_dir }}
+LimitCORE=infinity
+Environment=LPCONFIG=launchpad-appserver
+SyslogIdentifier=launchpad
+ExecStart={{ code_dir }}/bin/run -i ${LPCONFIG}
+ExecReload=/bin/kill -HUP $MAINPID
+KillMode=mixed
+Restart=on-failure
+# gunicorn is configured to gracefully shut down over two minutes. Allow a
+# few more seconds, then kill it if necessary.
+TimeoutStopSec=125
+PrivateTmp=true
+
+[Install]
+WantedBy=multi-user.target
+
diff --git a/charm/launchpad-appserver/templates/logrotate.conf.j2 b/charm/launchpad-appserver/templates/logrotate.conf.j2
new file mode 100644
index 0000000..494f246
--- /dev/null
+++ b/charm/launchpad-appserver/templates/logrotate.conf.j2
@@ -0,0 +1,16 @@
+{{ logs_dir }}/launchpad.log
+{
+ rotate 21
+ daily
+ dateext
+ delaycompress
+ compress
+ notifempty
+ missingok
+ create 0644 syslog adm
+ sharedscripts
+ postrotate
+ invoke-rc.d rsyslog rotate >/dev/null
+ endscript
+}
+
diff --git a/charm/launchpad-appserver/templates/rsyslog.j2 b/charm/launchpad-appserver/templates/rsyslog.j2
new file mode 100644
index 0000000..b62973e
--- /dev/null
+++ b/charm/launchpad-appserver/templates/rsyslog.j2
@@ -0,0 +1,10 @@
+template(name="talisker" type="list") {
+ property(name="msg" position.from="2")
+ constant(value="\n")
+}
+
+if $programname == "launchpad" then {
+ action(type="omfile" file="{{ logs_dir }}/launchpad.log" template="talisker" fileOwner="syslog" fileGroup="adm")
+ stop
+}
+
diff --git a/charm/layer/launchpad-base/config.yaml b/charm/layer/launchpad-base/config.yaml
index adc4e2c..cb665fe 100644
--- a/charm/layer/launchpad-base/config.yaml
+++ b/charm/layer/launchpad-base/config.yaml
@@ -1,4 +1,56 @@
options:
+ cron_control_url:
+ type: string
+ description: URL of file used to control whether cron scripts run.
+ default: "file:cronscripts.ini"
+ db_primary:
+ type: string
+ description: Connection string for primary database.
+ default: ""
+ db_session:
+ type: string
+ description: Connection string for session database.
+ default: ""
+ db_standby:
+ type: string
+ description: Connection string for standby database, if any.
+ default: ""
+ domain:
+ type: string
+ description: Domain name for this instance.
+ default: "launchpad.test"
+ domain_xmlrpc_private:
+ type: string
+ description: Domain name for this instance's private XML-RPC service.
+ default: "xmlrpc-private.launchpad.test"
+ http_proxy:
+ type: string
+ description: Proxy to be used when issuing HTTP requests.
+ default: ""
+ log_hosts_allow:
+ type: string
+ description: >
+ Hosts that should be allowed to rsync logs. Note that this relies on
+ the nrpe subordinate charm to set up /etc/rsyncd.conf properly.
+ default: ""
+ oops_prefix:
+ type: string
+ description: A prefix for OOPS codes for this instance.
+ default: "TEST"
+ openid_alternate_provider_roots:
+ type: string
+ description: >
+ Comma-separated list of additional URLs to recognise as valid prefixes
+ for our accounts' OpenID identifiers.
+ default: ""
+ openid_provider_root:
+ type: string
+ description: URL to the OpenID provider to authenticate against.
+ default: https://testopenid.test/
+ session_cookie_name:
+ type: string
+ description: The ID of the cookie used to store the session token.
+ default: launchpad_dev
# layer-apt
install_sources:
default: |
diff --git a/charm/layer/launchpad-base/lib/charms/launchpad/base.py b/charm/layer/launchpad-base/lib/charms/launchpad/base.py
new file mode 100644
index 0000000..67bfdcb
--- /dev/null
+++ b/charm/layer/launchpad-base/lib/charms/launchpad/base.py
@@ -0,0 +1,82 @@
+# Copyright 2022 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+import os.path
+
+from charmhelpers.core import hookenv, host, templating
+from ols import base
+
+
+def home_dir():
+ return os.path.join("/home", base.user())
+
+
+def oopses_dir():
+ return os.path.join(base.base_dir(), "oopses")
+
+
+def secrets_dir():
+ return os.path.join(base.base_dir(), "secrets")
+
+
+def var_dir():
+ return os.path.join(base.base_dir(), "var")
+
+
+def ensure_lp_directories():
+ for dirpath in oopses_dir(), var_dir():
+ host.mkdir(dirpath, group=base.user(), perms=0o775)
+ host.mkdir(secrets_dir(), group=base.user(), perms=0o750)
+ host.mkdir(home_dir(), owner=base.user(), group=base.user(), perms=0o755)
+
+
+def get_service_config():
+ config = dict(hookenv.config())
+ config.update(
+ {
+ "base_dir": base.base_dir(),
+ "code_dir": base.code_dir(),
+ "logs_dir": base.logs_dir(),
+ "oopses_dir": oopses_dir(),
+ "secrets_dir": secrets_dir(),
+ "user": base.user(),
+ "var_dir": var_dir(),
+ }
+ )
+ return config
+
+
+def config_file_path(name, secret=False):
+ if secret:
+ config_dir = os.path.join(base.base_dir(), "secrets")
+ else:
+ config_dir = os.path.join(base.code_dir(), "production-configs")
+ return os.path.join(config_dir, name)
+
+
+def configure_lazr(config, template, name, secret=False):
+ hookenv.log("Writing service configuration.")
+ templating.render(
+ template,
+ config_file_path(name, secret=secret),
+ config,
+ owner="root",
+ group=base.user(),
+ perms=0o440 if secret else 0o444,
+ )
+
+
+def lazr_config_files():
+ return [
+ config_file_path("launchpad-base-lazr.conf"),
+ config_file_path("launchpad-base-secrets-lazr.conf", secret=True),
+ ]
+
+
+def configure_rsync(config, template, name):
+ hookenv.log("Writing rsync configuration.")
+ rsync_path = os.path.join("/etc/rsync-juju.d", name)
+ if config["log_hosts_allow"]:
+ templating.render(template, rsync_path, config, perms=0o644)
+ elif os.path.exists(rsync_path):
+ os.unlink(rsync_path)
diff --git a/charm/layer/launchpad-base/reactive/launchpad-base.py b/charm/layer/launchpad-base/reactive/launchpad-base.py
index 88a6f5b..e5e90ea 100644
--- a/charm/layer/launchpad-base/reactive/launchpad-base.py
+++ b/charm/layer/launchpad-base/reactive/launchpad-base.py
@@ -3,7 +3,13 @@
import subprocess
-from charms.reactive import remove_state, when
+from charms.launchpad.base import (
+ configure_lazr,
+ configure_rsync,
+ ensure_lp_directories,
+ get_service_config,
+)
+from charms.reactive import remove_state, set_state, when, when_not
from ols import base
@@ -19,13 +25,39 @@ def create_virtualenv(wheels_dir, codedir, python_exe):
base.create_virtualenv = create_virtualenv
+@when("ols.configured")
+@when_not("launchpad.base.configured")
+def configure():
+ ensure_lp_directories()
+ config = get_service_config()
+ # XXX cjwatson 2022-09-07: Some config items have no reasonable default.
+ # We should set the workload status to blocked in that case.
+ configure_lazr(
+ config,
+ "launchpad-base-lazr.conf",
+ "launchpad-base-lazr.conf",
+ )
+ configure_lazr(
+ config,
+ "launchpad-base-secrets-lazr.conf",
+ "launchpad-base-secrets-lazr.conf",
+ secret=True,
+ )
+ configure_rsync(
+ config, "launchpad-base-rsync.conf", "010-launchpad-base.conf"
+ )
+ set_state("launchpad.base.configured")
+
+
@when("config.changed.build_label")
def build_label_changed():
remove_state("ols.service.installed")
remove_state("ols.configured")
+ remove_state("launchpad.base.configured")
remove_state("service.configured")
@when("config.changed")
def config_changed():
+ remove_state("launchpad.base.configured")
remove_state("service.configured")
diff --git a/charm/layer/launchpad-base/templates/launchpad-base-lazr.conf b/charm/layer/launchpad-base/templates/launchpad-base-lazr.conf
new file mode 100644
index 0000000..f0b4aeb
--- /dev/null
+++ b/charm/layer/launchpad-base/templates/launchpad-base-lazr.conf
@@ -0,0 +1,71 @@
+# Public configuration data. The contents of this file may be freely shared
+# with developers if needed for debugging.
+
+# A schema's sections, keys, and values are automatically inherited, except
+# for '.optional' sections. Update this config to override key values.
+# Values are strings, except for numbers that look like ints. The tokens
+# true, false, and none are treated as True, False, and None.
+
+[meta]
+extends: ../lib/lp/services/config/schema-lazr.conf
+
+[canonical]
+bounce_address: noreply@xxxxxxxxxxxxx
+pid_dir: {{ var_dir }}
+
+[database]
+db_statement_timeout: 15000
+rw_main_primary: {{ db_primary }}
+{%- if db_standby %}
+rw_main_standby: {{ db_standby }}
+{%- endif %}
+soft_request_timeout: 8000
+
+[error_reports]
+error_dir: {{ oopses_dir }}
+oops_prefix: {{ oops_prefix }}
+
+[launchpad]
+config_overlay_dir: {{ secrets_dir }}
+http_proxy: {{ http_proxy or "none" }}
+openid_alternate_provider_roots: {{ openid_alternate_provider_roots or "none" }}
+openid_provider_root: {{ openid_provider_root }}
+
+[launchpad_session]
+database: {{ db_session }}
+cookie: {{ session_cookie_name }}
+
+[vhost.mainsite]
+hostname: {{ domain }}
+althostnames: localhost, www.{{ domain }}
+openid_delegate_profile: true
+
+[vhost.answers]
+hostname: answers.{{ domain }}
+
+[vhost.api]
+hostname: api.{{ domain }}
+
+[vhost.blueprints]
+hostname: blueprints.{{ domain }}
+
+[vhost.bugs]
+hostname: bugs.{{ domain }}
+
+[vhost.code]
+hostname: code.{{ domain }}
+
+[vhost.feeds]
+hostname: feeds.{{ domain }}
+# Don't link to HTTPS for feeds.
+rooturl: http://feeds.{{ domain }}/
+
+[vhost.translations]
+hostname: translations.{{ domain }}
+
+[vhost.xmlrpc]
+hostname: xmlrpc.{{ domain }}
+
+[vhost.xmlrpc_private]
+hostname: {{ domain_xmlrpc_private }}
+
diff --git a/charm/layer/launchpad-base/templates/launchpad-base-rsync.conf b/charm/layer/launchpad-base/templates/launchpad-base-rsync.conf
new file mode 100644
index 0000000..2921300
--- /dev/null
+++ b/charm/layer/launchpad-base/templates/launchpad-base-rsync.conf
@@ -0,0 +1,8 @@
+
+[lp-logs]
+ path = {{ logs_dir }}
+ comment = LP Logs
+ list = false
+ read only = true
+ hosts allow = {{ log_hosts_allow }}
+
diff --git a/charm/layer/launchpad-base/templates/launchpad-base-secrets-lazr.conf b/charm/layer/launchpad-base/templates/launchpad-base-secrets-lazr.conf
new file mode 100644
index 0000000..7229414
--- /dev/null
+++ b/charm/layer/launchpad-base/templates/launchpad-base-secrets-lazr.conf
@@ -0,0 +1,9 @@
+# Secret configuration data. This is stored in an overlay directory, mainly
+# to avoid accidental information leaks from the public configuration file.
+# Entries in this file should not be shared with developers, although the
+# structure of the file is not secret, only configuration values.
+
+# A schema's sections, keys, and values are automatically inherited, except
+# for '.optional' sections. Update this config to override key values.
+# Values are strings, except for numbers that look like ints. The tokens
+# true, false, and none are treated as True, False, and None.
