launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #28188
[Merge] ~cjwatson/lpbuildbot-worker:ops-charm into lpbuildbot-worker:main
Colin Watson has proposed merging ~cjwatson/lpbuildbot-worker:ops-charm into lpbuildbot-worker:main.
Commit message:
Add a charm to set up a buildbot worker node
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~cjwatson/lpbuildbot-worker/+git/lpbuildbot-worker/+merge/416408
This essentially just automates the existing steps that were used to set up Launchpad's current worker nodes. The tests are a bit on the basic side, but they more or less work, and they were an opportunity to experiment with testing operator charms using `pytest`.
The `juju deploy` command in the README won't work until the charm has been built by a Launchpad charm recipe and pushed to Charmhub, but it gives an idea of the intended workflow.
--
Your team Launchpad code reviewers is requested to review the proposed merge of ~cjwatson/lpbuildbot-worker:ops-charm into lpbuildbot-worker:main.
diff --git a/charm/.gitignore b/charm/.gitignore
new file mode 100644
index 0000000..7a9b421
--- /dev/null
+++ b/charm/.gitignore
@@ -0,0 +1,7 @@
+venv/
+build/
+*.charm
+
+__pycache__/
+*.py[cod]
+.tox
diff --git a/charm/.jujuignore b/charm/.jujuignore
new file mode 100644
index 0000000..a130358
--- /dev/null
+++ b/charm/.jujuignore
@@ -0,0 +1,4 @@
+/venv
+*.py[cod]
+*.charm
+.tox
diff --git a/charm/LICENSE b/charm/LICENSE
new file mode 100644
index 0000000..d645695
--- /dev/null
+++ b/charm/LICENSE
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/charm/README.md b/charm/README.md
new file mode 100644
index 0000000..fd49106
--- /dev/null
+++ b/charm/README.md
@@ -0,0 +1,29 @@
+# charm
+
+## Description
+
+Launchpad runs its continuous integration using
+[Buildbot](https://buildbot.net/). This charm provides a suitable worker
+node.
+
+This charm is highly specialized for use by Launchpad's CI system, and will
+not work for other purposes.
+
+## Usage
+
+ juju deploy ch:lpbuildbot-worker
+
+Until such time as a manager relation is added, you'll need to set
+`manager-host` to the host name of the buildbot manager and
+`buildbot-password` to the password used by the manager to contact this
+worker.
+
+## Relations
+
+None at present (though a relation with a buildbot manager may be added in
+future).
+
+## Contributing
+
+Please see the [Juju SDK docs](https://juju.is/docs/sdk) for guidelines
+on enhancements to this charm following best practice guidelines.
diff --git a/charm/charmcraft.yaml b/charm/charmcraft.yaml
new file mode 100644
index 0000000..f128aad
--- /dev/null
+++ b/charm/charmcraft.yaml
@@ -0,0 +1,7 @@
+type: "charm"
+parts:
+ charm:
+ charm-python-packages: [setuptools]
+bases:
+ - name: "ubuntu"
+ channel: "18.04"
diff --git a/charm/config.yaml b/charm/config.yaml
new file mode 100644
index 0000000..abf0721
--- /dev/null
+++ b/charm/config.yaml
@@ -0,0 +1,25 @@
+# Copyright 2021 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+options:
+ ubuntu-series:
+ type: string
+ default: xenial
+ description: >
+ Space-separated list of Ubuntu series for which to maintain workers.
+ manager-host:
+ type: string
+ default:
+ description: Buildbot manager host name.
+ manager-port:
+ type: int
+ default: 9989
+ description: Buildbot manager port.
+ buildbot-password:
+ type: string
+ default:
+ description: Password used to contact the Buildbot manager.
+ active:
+ type: boolean
+ default: true
+ description: If true, start Buildbot workers.
diff --git a/charm/metadata.yaml b/charm/metadata.yaml
new file mode 100644
index 0000000..9021930
--- /dev/null
+++ b/charm/metadata.yaml
@@ -0,0 +1,11 @@
+# Copyright 2021 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+name: lpbuildbot-worker
+display-name: Launchpad Buildbot worker node
+summary: A preconfigured Launchpad Buildbot worker node.
+description: >
+ Launchpad runs its continuous integration using Buildbot. This charm
+ provides a suitable worker node.
+series:
+ - bionic
diff --git a/charm/requirements-dev.txt b/charm/requirements-dev.txt
new file mode 100644
index 0000000..8fef638
--- /dev/null
+++ b/charm/requirements-dev.txt
@@ -0,0 +1,4 @@
+-r requirements.txt
+pyfakefs
+pytest
+pytest-subprocess
diff --git a/charm/requirements.txt b/charm/requirements.txt
new file mode 100644
index 0000000..e352ba9
--- /dev/null
+++ b/charm/requirements.txt
@@ -0,0 +1,2 @@
+jinja2
+ops >= 1.2.0
diff --git a/charm/src/charm.py b/charm/src/charm.py
new file mode 100755
index 0000000..c8418c9
--- /dev/null
+++ b/charm/src/charm.py
@@ -0,0 +1,294 @@
+#!/usr/bin/env python3
+# Copyright 2021-2022 Canonical Ltd.
+# See LICENSE file for licensing details.
+
+"""Install and configure a Launchpad buildbot worker."""
+
+import grp
+import json
+import logging
+import os
+import pwd
+import shutil
+import subprocess
+from pathlib import Path
+
+from jinja2 import Environment, FileSystemLoader
+from ops.charm import CharmBase
+from ops.framework import StoredState
+from ops.main import main
+from ops.model import ActiveStatus, BlockedStatus, MaintenanceStatus
+
+logger = logging.getLogger(__name__)
+
+
+class BlockedOnConfig(Exception):
+ def __init__(self, name):
+ super().__init__("Waiting for {} to be set".format(name))
+
+
+class LPBuildbotWorkerCharm(CharmBase):
+
+ _stored = StoredState()
+
+ def __init__(self, *args):
+ super().__init__(*args)
+ self.framework.observe(self.on.install, self._on_install)
+ self.framework.observe(self.on.upgrade_charm, self._on_install)
+ self.framework.observe(self.on.config_changed, self._on_config_changed)
+ self._stored.set_default(
+ ubuntu_series=set(),
+ manager_host=None,
+ manager_port=None,
+ buildbot_password=None,
+ )
+
+ def _set_maintenance_step(self, description):
+ self.unit.status = MaintenanceStatus(description)
+ logger.info(description)
+
+ def _run_as_buildbot(self, args, **kwargs):
+ return subprocess.run(["sudo", "-Hu", "buildbot"] + args, **kwargs)
+
+ def _require_config(self, name):
+ value = self.config.get(name)
+ if not value:
+ raise BlockedOnConfig(name)
+ return value
+
+ def _list_lxd_images(self):
+ images = json.loads(
+ self._run_as_buildbot(
+ ["lxc", "image", "list", "-f", "json"],
+ stdout=subprocess.PIPE,
+ check=True,
+ universal_newlines=True,
+ ).stdout
+ )
+ names = []
+ for image in images:
+ for alias in image["aliases"]:
+ names.append(alias["name"])
+ return names
+
+ def _chown(self, path, user, group):
+ uid = pwd.getpwnam(user).pw_uid
+ gid = grp.getgrnam(group).gr_gid
+ os.chown(str(path), uid, gid)
+
+ def _render_template(
+ self, source, target, context, user="root", group="root", mode=0o644
+ ):
+ template_env = Environment(
+ loader=FileSystemLoader(str(self.charm_dir / "templates"))
+ )
+ template = template_env.get_template(source)
+ content = template.render(context)
+ target = Path(target)
+ if not target.parent.exists():
+ target.parent.mkdir(mode=0o755, parents=True)
+ self._chown(target.parent, user, group)
+ target.write_text(content)
+ self._chown(target, user, group)
+ os.chmod(str(target), mode)
+
+ def _install_lpbuildbot_worker(self):
+ self._set_maintenance_step("Installing lpbuildbot-worker")
+ subprocess.run(
+ ["add-apt-repository", "-y", "ppa:launchpad/ubuntu/ppa"],
+ check=True,
+ )
+ subprocess.run(
+ ["apt-get", "-y", "install", "lpbuildbot-worker"], check=True
+ )
+
+ def _install_lxd(self):
+ if Path("/usr/bin/lxc").exists():
+ self._set_maintenance_step("Removing lxd .debs")
+ subprocess.run(
+ [
+ "apt-get",
+ "-y",
+ "purge",
+ "lxc-common",
+ "lxcfs",
+ "lxd-client",
+ ],
+ check=True,
+ )
+ # We may need to delete this leftover interface in order for
+ # "lxd init --auto" to succeed after installing the snap. It's
+ # fine if it doesn't exist.
+ if (
+ subprocess.run(
+ ["ip", "link", "show", "dev", "lxdbr0"],
+ stdout=subprocess.DEVNULL,
+ stderr=subprocess.DEVNULL,
+ ).returncode
+ == 0
+ ):
+ subprocess.run(
+ ["ip", "link", "delete", "dev", "lxdbr0"], check=True
+ )
+
+ if not Path("/snap/bin/lxc").exists():
+ self._set_maintenance_step("Installing lxd snap")
+ subprocess.run(["snap", "install", "lxd"], check=True)
+
+ if not Path("/var/snap/lxd/common/lxd/server.key").exists():
+ self._set_maintenance_step("Initializing lxd")
+ subprocess.run(
+ ["lxd", "init", "--auto", "--storage-backend=zfs"], check=True
+ )
+
+ def _configure_buildbot_user(self):
+ self._set_maintenance_step("Configuring buildbot user")
+ subprocess.run(["adduser", "buildbot", "lxd"], check=True)
+ # Snaps don't currently work well with users whose home directories
+ # aren't under /home.
+ old_home = Path("/var/lib/buildbot")
+ new_home = Path("/home/buildbot")
+ if old_home.exists() and not old_home.is_symlink():
+ shutil.move(str(old_home), str(new_home))
+ old_home.symlink_to(new_home)
+ subprocess.run(
+ ["usermod", "-d", str(new_home), "buildbot"], check=True
+ )
+ ssh_key_path = new_home / ".ssh" / "launchpad_lxd_id_rsa"
+ if not ssh_key_path.exists():
+ self._run_as_buildbot(
+ ["mkdir", "-m700", "-p", str(ssh_key_path.parent)], check=True
+ )
+ self._run_as_buildbot(
+ [
+ "ssh-keygen",
+ "-t",
+ "rsa",
+ "-b",
+ "2048",
+ "-N",
+ "",
+ "-f",
+ str(ssh_key_path),
+ ],
+ check=True,
+ )
+
+ def _make_workers(self):
+ ubuntu_series = set(self._require_config("ubuntu-series").split())
+ manager_host = self._require_config("manager-host")
+ manager_port = self._require_config("manager-port")
+ buildbot_password = self._require_config("buildbot-password")
+
+ workers_path = Path("/home/buildbot/workers")
+ if not workers_path.exists():
+ self._run_as_buildbot(
+ ["mkdir", "-m755", "-p", str(workers_path)], check=True
+ )
+
+ current_images = self._list_lxd_images()
+ for series in ubuntu_series:
+ self._set_maintenance_step("Making worker for {}".format(series))
+ base_path = workers_path / "{}-lxd-worker".format(series)
+ if not base_path.exists():
+ self._run_as_buildbot(
+ ["mkdir", "-m755", "-p", str(base_path)], check=True
+ )
+ lp_path = base_path / "devel"
+ dependencies_path = base_path / "dependencies"
+ sourcecode_path = dependencies_path / "sourcecode"
+ download_cache_path = dependencies_path / "download-cache"
+ if not lp_path.exists():
+ self._run_as_buildbot(
+ [
+ "git",
+ "clone",
+ "https://git.launchpad.net/launchpad";,
+ str(lp_path),
+ ],
+ check=True,
+ )
+ if not sourcecode_path.exists():
+ self._run_as_buildbot(
+ ["mkdir", "-m755", "-p", str(sourcecode_path)], check=True
+ )
+ if not download_cache_path.exists():
+ self._run_as_buildbot(
+ [
+ "git",
+ "clone",
+ "https://git.launchpad.net/lp-source-dependencies";,
+ str(download_cache_path),
+ ],
+ check=True,
+ )
+ if "lptests-{}".format(series) not in current_images:
+ self._run_as_buildbot(
+ ["create-lp-tests-lxd", series, str(base_path)], check=True
+ )
+
+ self._render_template(
+ "buildbot.tac.j2",
+ str(base_path / "buildbot.tac"),
+ {
+ "buildbot_password": buildbot_password,
+ "manager_host": manager_host,
+ "manager_port": manager_port,
+ "name": self.unit.name.replace("/", "-"),
+ "series": series,
+ },
+ user="buildbot",
+ group="buildbot",
+ mode=0o640,
+ )
+ service_name = "buildbot-worker@{}-lxd-worker.service".format(
+ series
+ )
+ if self.config.get("active", True):
+ subprocess.run(
+ ["systemctl", "enable", service_name], check=True
+ )
+ subprocess.run(
+ ["systemctl", "restart", service_name], check=True
+ )
+ else:
+ subprocess.run(["systemctl", "disable", service_name])
+ subprocess.run(["systemctl", "stop", service_name])
+
+ for image in current_images:
+ if not image.startswith("lptests-"):
+ continue
+ series = image[len("lptests-") :]
+ if series not in ubuntu_series:
+ self._set_maintenance_step(
+ "Deleting obsolete worker for {}".format(series)
+ )
+ self._run_as_buildbot(
+ ["lxc", "image", "delete", image], check=True
+ )
+ service_name = "buildbot-worker@{}-lxd-worker.service".format(
+ series
+ )
+ subprocess.run(["systemctl", "disable", service_name])
+ subprocess.run(["systemctl", "stop", service_name])
+
+ self._stored.ubuntu_series = ubuntu_series
+
+ def _on_install(self, event):
+ self._install_lpbuildbot_worker()
+ self._install_lxd()
+ self._configure_buildbot_user()
+ self._on_config_changed(event)
+
+ def _on_config_changed(self, _):
+ try:
+ self._make_workers()
+ except BlockedOnConfig as e:
+ self.unit.status = BlockedStatus(str(e))
+ else:
+ self.unit.status = ActiveStatus()
+ logger.info("Ready")
+
+
+if __name__ == "__main__":
+ main(LPBuildbotWorkerCharm)
diff --git a/charm/templates/buildbot.tac.j2 b/charm/templates/buildbot.tac.j2
new file mode 100644
index 0000000..eaa0a61
--- /dev/null
+++ b/charm/templates/buildbot.tac.j2
@@ -0,0 +1,33 @@
+import os
+
+from buildbot_worker.bot import Worker
+from twisted.application import service
+from twisted.python.log import (
+ FileLogObserver,
+ ILogObserver,
+ )
+from twisted.python.logfile import LogFile
+
+
+basedir = '/home/buildbot/workers/{{ series }}-lxd-worker'
+manager_host = '{{ manager_host }}'
+manager_port = {{ manager_port }}
+name = '{{ name }}'
+password = '{{ buildbot_password }}'
+keepalive = 600
+umask = 0o22
+rotateLength = 1000000
+maxRotatedFiles = None
+
+# note: this line is matched against to check that this is a worker
+# directory; do not edit it.
+application = service.Application('buildbot-worker')
+
+logfile = LogFile.fromFullPath(
+ os.path.join(basedir, 'twistd.log'), rotateLength=rotateLength,
+ maxRotatedFiles=maxRotatedFiles)
+application.setComponent(ILogObserver, FileLogObserver(logfile).emit)
+w = Worker(
+ manager_host, manager_port, name, password, basedir, keepalive,
+ umask=umask)
+w.setServiceParent(application)
diff --git a/charm/tests/__init__.py b/charm/tests/__init__.py
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/charm/tests/__init__.py
diff --git a/charm/tests/test_charm.py b/charm/tests/test_charm.py
new file mode 100644
index 0000000..314e7e0
--- /dev/null
+++ b/charm/tests/test_charm.py
@@ -0,0 +1,392 @@
+# Copyright 2021-2022 Canonical Ltd.
+# See LICENSE file for licensing details.
+#
+# Learn more about testing at: https://juju.is/docs/sdk/testing
+
+import json
+import os
+from pathlib import Path
+
+import pytest
+from ops.model import ActiveStatus, MaintenanceStatus
+from ops.testing import Harness
+
+from charm import BlockedOnConfig, LPBuildbotWorkerCharm
+
+# The "fs" fixture is a fake filesystem from pyfakefs; the "fp" fixture is
+# from pytest-subprocess.
+
+
+class FakePasswd:
+ def __init__(self, pw_uid):
+ self.pw_uid = pw_uid
+
+
+class FakeGroup:
+ def __init__(self, gr_gid):
+ self.gr_gid = gr_gid
+
+
+@pytest.fixture
+def fake_user(monkeypatch):
+ monkeypatch.setattr("pwd.getpwnam", lambda user: FakePasswd(1000))
+ monkeypatch.setattr("grp.getgrnam", lambda user: FakeGroup(1000))
+
+
+@pytest.fixture
+def harness():
+ harness = Harness(LPBuildbotWorkerCharm)
+ harness.begin()
+ yield harness
+ harness.cleanup()
+
+
+def test_install_lpbuildbot_worker(harness, fp):
+ fp.keep_last_process(True)
+ fp.register([fp.any()])
+
+ harness.charm._install_lpbuildbot_worker()
+
+ assert harness.model.unit.status == MaintenanceStatus(
+ "Installing lpbuildbot-worker"
+ )
+ assert list(fp.calls) == [
+ ["add-apt-repository", "-y", "ppa:launchpad/ubuntu/ppa"],
+ ["apt-get", "-y", "install", "lpbuildbot-worker"],
+ ]
+
+
+def test_install_lxd_removes_debs_and_installs_snap(harness, fs, fp):
+ Path("/usr/bin").mkdir(parents=True)
+ Path("/usr/bin/lxc").touch()
+ fp.keep_last_process(True)
+ fp.register([fp.any()])
+
+ harness.charm._install_lxd()
+
+ assert harness.model.unit.status == MaintenanceStatus("Initializing lxd")
+ assert list(fp.calls) == [
+ ["apt-get", "-y", "purge", "lxc-common", "lxcfs", "lxd-client"],
+ ["ip", "link", "show", "dev", "lxdbr0"],
+ ["ip", "link", "delete", "dev", "lxdbr0"],
+ ["snap", "install", "lxd"],
+ ["lxd", "init", "--auto", "--storage-backend=zfs"],
+ ]
+
+
+def test_install_lxd_snap_already_installed(harness, fs, fp):
+ Path("/snap/bin").mkdir(parents=True)
+ Path("/snap/bin/lxc").touch()
+ fp.keep_last_process(True)
+ fp.register([fp.any()])
+
+ harness.charm._install_lxd()
+
+ assert harness.model.unit.status == MaintenanceStatus("Initializing lxd")
+ assert list(fp.calls) == [
+ ["lxd", "init", "--auto", "--storage-backend=zfs"],
+ ]
+
+
+def test_install_lxd_already_configured(harness, fs, fp):
+ Path("/snap/bin").mkdir(parents=True)
+ Path("/snap/bin/lxc").touch()
+ Path("/var/snap/lxd/common/lxd").mkdir(parents=True)
+ Path("/var/snap/lxd/common/lxd/server.key").touch()
+ fp.keep_last_process(True)
+ fp.register([fp.any()])
+
+ harness.charm._install_lxd()
+
+ assert list(fp.calls) == []
+
+
+def test_configure_buildbot_user_moves_home_directory(harness, fs, fp):
+ Path("/var/lib/buildbot").mkdir(parents=True)
+ Path("/home").mkdir()
+ fp.keep_last_process(True)
+ fp.register([fp.any()])
+
+ harness.charm._configure_buildbot_user()
+
+ assert list(fp.calls) == [
+ ["adduser", "buildbot", "lxd"],
+ ["usermod", "-d", "/home/buildbot", "buildbot"],
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "mkdir",
+ "-m700",
+ "-p",
+ "/home/buildbot/.ssh",
+ ],
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "ssh-keygen",
+ "-t",
+ "rsa",
+ "-b",
+ "2048",
+ "-N",
+ "",
+ "-f",
+ "/home/buildbot/.ssh/launchpad_lxd_id_rsa",
+ ],
+ ]
+ assert Path("/home/buildbot").is_dir()
+ assert not Path("/home/buildbot").is_symlink()
+ assert Path("/var/lib/buildbot").is_symlink()
+ assert os.readlink("/var/lib/buildbot") == "/home/buildbot"
+
+
+def test_configure_buildbot_user_keeps_moved_home_directory(harness, fs, fp):
+ Path("/home/buildbot").mkdir(parents=True)
+ Path("/var/lib").mkdir(parents=True)
+ Path("/var/lib/buildbot").symlink_to("/home/buildbot")
+ fp.keep_last_process(True)
+ fp.register([fp.any()])
+
+ harness.charm._configure_buildbot_user()
+
+ assert list(fp.calls) == [
+ ["adduser", "buildbot", "lxd"],
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "mkdir",
+ "-m700",
+ "-p",
+ "/home/buildbot/.ssh",
+ ],
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "ssh-keygen",
+ "-t",
+ "rsa",
+ "-b",
+ "2048",
+ "-N",
+ "",
+ "-f",
+ "/home/buildbot/.ssh/launchpad_lxd_id_rsa",
+ ],
+ ]
+ assert Path("/home/buildbot").is_dir()
+ assert not Path("/home/buildbot").is_symlink()
+ assert Path("/var/lib/buildbot").is_symlink()
+ assert os.readlink("/var/lib/buildbot") == "/home/buildbot"
+
+
+def test_configure_buildbot_user_keeps_existing_ssh_key(harness, fs, fp):
+ Path("/home/buildbot/.ssh").mkdir(parents=True)
+ Path("/home/buildbot/.ssh/launchpad_lxd_id_rsa").write_text("key")
+ fp.keep_last_process(True)
+ fp.register([fp.any()])
+
+ harness.charm._configure_buildbot_user()
+
+ assert list(fp.calls) == [["adduser", "buildbot", "lxd"]]
+ assert (
+ Path("/home/buildbot/.ssh/launchpad_lxd_id_rsa").read_text() == "key"
+ )
+
+
+@pytest.mark.parametrize(
+ "empty_key",
+ ["ubuntu-series", "manager-host", "manager-port", "buildbot-password"],
+)
+def test_make_workers_requires_config(harness, fs, fp, empty_key):
+ # ubuntu-series and manager-port have defaults in config.yaml.
+ config = {
+ "manager-host": "manager.example.com",
+ "buildbot-password": "secret",
+ }
+ config[empty_key] = ""
+ harness._update_config(config)
+
+ pytest.raises(BlockedOnConfig, harness.charm._make_workers)
+
+
+def test_make_workers(harness, fs, fp, fake_user):
+ fs.add_real_directory(harness.charm.charm_dir / "templates")
+ fp.keep_last_process(True)
+ fp.register(
+ ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"],
+ stdout=json.dumps({}),
+ )
+ fp.register([fp.any()])
+ harness._update_config(
+ {"manager-host": "manager.example.com", "buildbot-password": "secret"}
+ )
+
+ harness.charm._make_workers()
+
+ assert harness.model.unit.status == MaintenanceStatus(
+ "Making worker for xenial"
+ )
+ assert list(fp.calls) == [
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "mkdir",
+ "-m755",
+ "-p",
+ "/home/buildbot/workers",
+ ],
+ ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"],
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "mkdir",
+ "-m755",
+ "-p",
+ "/home/buildbot/workers/xenial-lxd-worker",
+ ],
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "git",
+ "clone",
+ "https://git.launchpad.net/launchpad";,
+ "/home/buildbot/workers/xenial-lxd-worker/devel",
+ ],
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "mkdir",
+ "-m755",
+ "-p",
+ "/home/buildbot/workers/xenial-lxd-worker/dependencies/sourcecode",
+ ],
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "git",
+ "clone",
+ "https://git.launchpad.net/lp-source-dependencies";,
+ "/home/buildbot/workers/xenial-lxd-worker/dependencies/"
+ "download-cache",
+ ],
+ [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "create-lp-tests-lxd",
+ "xenial",
+ "/home/buildbot/workers/xenial-lxd-worker",
+ ],
+ ["systemctl", "enable", "buildbot-worker@xenial-lxd-worker.service"],
+ ["systemctl", "restart", "buildbot-worker@xenial-lxd-worker.service"],
+ ]
+ buildbot_tac = (
+ Path("/home/buildbot/workers/xenial-lxd-worker/buildbot.tac")
+ .read_text()
+ .splitlines()
+ )
+ assert (
+ "basedir = '/home/buildbot/workers/xenial-lxd-worker'" in buildbot_tac
+ )
+ assert "manager_host = 'manager.example.com'" in buildbot_tac
+ assert "manager_port = 9989" in buildbot_tac
+ assert "password = 'secret'" in buildbot_tac
+ assert harness.charm._stored.ubuntu_series == {"xenial"}
+
+
+def test_make_workers_deletes_obsolete_workers(harness, fs, fp, fake_user):
+ fs.add_real_directory(harness.charm.charm_dir / "templates")
+ fp.keep_last_process(True)
+ fp.register(
+ ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"],
+ stdout=json.dumps([{"aliases": [{"name": "lptests-precise"}]}]),
+ )
+ fp.register([fp.any()])
+ harness._update_config(
+ {"manager-host": "manager.example.com", "buildbot-password": "secret"}
+ )
+
+ harness.charm._make_workers()
+
+ assert harness.model.unit.status == MaintenanceStatus(
+ "Deleting obsolete worker for precise"
+ )
+ assert [
+ "sudo",
+ "-Hu",
+ "buildbot",
+ "lxc",
+ "image",
+ "delete",
+ "lptests-precise",
+ ] in fp.calls
+ assert [
+ "systemctl",
+ "disable",
+ "buildbot-worker@precise-lxd-worker.service",
+ ] in fp.calls
+ assert [
+ "systemctl",
+ "stop",
+ "buildbot-worker@precise-lxd-worker.service",
+ ] in fp.calls
+
+
+def test_install(harness, fs, fp, fake_user):
+ fs.add_real_directory(harness.charm.charm_dir / "templates")
+ fp.keep_last_process(True)
+ fp.register(
+ ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"],
+ stdout=json.dumps({}),
+ )
+ fp.register([fp.any()])
+ harness._update_config(
+ {"manager-host": "manager.example.com", "buildbot-password": "secret"}
+ )
+
+ harness.charm.on.install.emit()
+
+ # Details are tested elsewhere; here we just ensure that install runs
+ # all the major steps.
+ assert ["apt-get", "-y", "install", "lpbuildbot-worker"] in fp.calls
+ assert ["lxd", "init", "--auto", "--storage-backend=zfs"] in fp.calls
+ assert ["adduser", "buildbot", "lxd"] in fp.calls
+ assert [
+ "systemctl",
+ "restart",
+ "buildbot-worker@xenial-lxd-worker.service",
+ ] in fp.calls
+ assert harness.model.unit.status == ActiveStatus()
+
+
+def test_config_changed(harness, fs, fp, fake_user):
+ fs.add_real_directory(harness.charm.charm_dir / "templates")
+ fp.keep_last_process(True)
+ fp.register(
+ ["sudo", "-Hu", "buildbot", "lxc", "image", "list", "-f", "json"],
+ stdout=json.dumps({}),
+ )
+ fp.register([fp.any()])
+
+ harness.update_config(
+ {
+ "manager-host": "manager.example.com",
+ "buildbot-password": "another-secret",
+ }
+ )
+
+ buildbot_tac = (
+ Path("/home/buildbot/workers/xenial-lxd-worker/buildbot.tac")
+ .read_text()
+ .splitlines()
+ )
+ assert "password = 'another-secret'" in buildbot_tac
diff --git a/charm/tox.ini b/charm/tox.ini
new file mode 100644
index 0000000..53ba640
--- /dev/null
+++ b/charm/tox.ini
@@ -0,0 +1,15 @@
+[tox]
+envlist =
+ py36
+ py37
+ py38
+ py39
+ py310
+# Charms aren't real Python packages, so we need a bit of hacking to make
+# tox work.
+skipsdist = True
+
+[testenv]
+deps = -r requirements-dev.txt
+setenv = PYTHONPATH={toxinidir}/src
+commands = pytest {posargs}
