yellow team mailing list archive
-
yellow team
-
Mailing list archive
-
Message #00354
[Merge] lp:~bac/charms/oneiric/buildbot-master/dynamic-relationship into lp:~yellow/charms/oneiric/buildbot-master/trunk
Brad Crittenden has proposed merging lp:~bac/charms/oneiric/buildbot-master/dynamic-relationship into lp:~yellow/charms/oneiric/buildbot-master/trunk.
Requested reviews:
Launchpad Yellow Squad (yellow): code
For more details, see:
https://code.launchpad.net/~bac/charms/oneiric/buildbot-master/dynamic-relationship/+merge/91737
Cleaned up helpers.py and local.py. Merged Graham's cleanup branch.
--
https://code.launchpad.net/~bac/charms/oneiric/buildbot-master/dynamic-relationship/+merge/91737
Your team Launchpad Yellow Squad is requested to review the proposed merge of lp:~bac/charms/oneiric/buildbot-master/dynamic-relationship into lp:~yellow/charms/oneiric/buildbot-master/trunk.
=== added file 'README.txt'
--- README.txt 1970-01-01 00:00:00 +0000
+++ README.txt 2012-02-06 23:20:22 +0000
@@ -0,0 +1,31 @@
+Demo of a single master.cfg file:
+
+This uses the standard buildbot Pyflakes example.
+
+Start with the buildbot-slave and buildbot-master charm in your charm
+repository.
+
+juju bootstrap
+juju deploy --repository=./charms local:buildbot-master
+juju deploy --repository=./charms local:buildbot-slave
+juju set buildbot-master extra-packages=git
+juju set buildbot-master config-file=`./charms/oneiric/buildbot-master/encode charms/oneiric/buildbot-master/examples/master.cfg`
+juju set buildbot-slave builders=runtests
+juju add-relation buildbot-slave buildbot-master
+
+Demo of a full buildbot master directory:
+
+juju bootstrap
+juju deploy --config=./charms/oneiric/buildbot-master/examples/lpbuildbot.yaml --repository=./charms local:buildbot-master
+juju deploy --config=./charms/oneiric/buildbot-slave/config.setuplxc.yaml --repository=./charms local:buildbot-slave
+juju add-relation buildbot-slave buildbot-master
+
+XXX Running setuplxc takes a long time. For now, then, replace the
+juju deploy of the slave with these two lines. This will mean that,
+when it works, the slave will connect to the master but when it tries
+to run tests it will fail (because the slave environment has not been
+set up).
+
+juju deploy --repository=./charms local:buildbot-slave
+juju set buildbot-slave builders=lucid_lp,lucid_db_lp
+
=== modified file 'config.yaml'
--- config.yaml 2012-02-01 15:39:49 +0000
+++ config.yaml 2012-02-06 23:20:22 +0000
@@ -13,10 +13,11 @@
default: /tmp/buildbot
config-file:
description: |
- A uuencoded master.cfg file. Use of this configuration is
- mutually exclusive with the use of config-transport,
- config-url. Use it like:
- juju set buildbot-master config-file `uuencode master.cfg`
+ An encoded master.cfg file. Use of this configuration is
+ mutually exclusive with the use of config-transport and
+ config-url. Use it with the `encode` executable in this
+ charm, like this:
+ juju set buildbot-master config-file=`./encode ./examples/master.cfg`
type: string
config-transport:
description: |
=== added file 'encode'
--- encode 1970-01-01 00:00:00 +0000
+++ encode 2012-02-06 23:20:22 +0000
@@ -0,0 +1,15 @@
+#!/usr/bin/python
+
+# encode examples/master.cfg | python -c 'import sys, base64; base64.decode(sys.stdin, sys.stdout)'
+
+import base64
+import StringIO
+import sys
+
+filename = sys.argv[1]
+tmp = StringIO.StringIO()
+with open(filename) as f:
+ base64.encode(f, tmp)
+tmp.flush()
+tmp.seek(0)
+sys.stdout.write(''.join(l.strip() for l in tmp.readlines()))
=== added file 'examples/master.cfg'
--- examples/master.cfg 1970-01-01 00:00:00 +0000
+++ examples/master.cfg 2012-02-06 23:20:22 +0000
@@ -0,0 +1,118 @@
+# -*- python -*-
+# ex: set syntax=python:
+
+# This is a sample buildmaster config file. It must be installed as
+# 'master.cfg' in your buildmaster's base directory.
+
+# This is the dictionary that the buildmaster pays attention to. We also use
+# a shorter alias to save typing.
+c = BuildmasterConfig = {}
+
+####### BUILDSLAVES
+
+# The 'slaves' list defines the set of recognized buildslaves. Each element is
+# a BuildSlave object, specifying a username and password. The same username and
+# password must be configured on the slave.
+from buildbot.buildslave import BuildSlave
+c['slaves'] = []
+
+# 'slavePortnum' defines the TCP port to listen on for connections from slaves.
+# This must match the value configured into the buildslaves (with their
+# --master option)
+c['slavePortnum'] = 9989
+
+####### CHANGESOURCES
+
+# the 'change_source' setting tells the buildmaster how it should find out
+# about source code changes. Here we point to the buildbot clone of pyflakes.
+
+from buildbot.changes.gitpoller import GitPoller
+c['change_source'] = GitPoller(
+ 'git://github.com/buildbot/pyflakes.git',
+ branch='master', pollinterval=1200)
+
+####### SCHEDULERS
+
+# Configure the Schedulers, which decide how to react to incoming changes. In this
+# case, just kick off a 'runtests' build
+
+from buildbot.scheduler import Scheduler
+c['schedulers'] = []
+c['schedulers'].append(Scheduler(name="all", branch=None,
+ treeStableTimer=None,
+ builderNames=["runtests"]))
+
+####### BUILDERS
+
+# The 'builders' list defines the Builders, which tell Buildbot how to perform a build:
+# what steps, and which slaves can execute them. Note that any particular build will
+# only take place on one slave.
+
+from buildbot.process.factory import BuildFactory
+from buildbot.steps.source import Git
+from buildbot.steps.shell import ShellCommand
+
+factory = BuildFactory()
+# check out the source
+factory.addStep(Git(repourl='git://github.com/buildbot/pyflakes.git', mode='copy'))
+# run the tests (note that this will require that 'trial' is installed)
+factory.addStep(ShellCommand(command=["trial", "pyflakes"]))
+
+from buildbot.config import BuilderConfig
+
+c['builders'] = [
+ BuilderConfig(name="runtests",
+ # Buildbot enforces that the slavenames list must not be empty. Our
+ # wrapper will remove the empty string and replace it with proper values.
+ slavenames=[''],
+ factory=factory),
+ ]
+
+####### STATUS TARGETS
+
+# 'status' is a list of Status Targets. The results of each build will be
+# pushed to these targets. buildbot/status/*.py has a variety to choose from,
+# including web pages, email senders, and IRC bots.
+
+c['status'] = []
+
+from buildbot.status import html
+from buildbot.status.web import auth, authz
+authz_cfg=authz.Authz(
+ # change any of these to True to enable; see the manual for more
+ # options
+ gracefulShutdown = False,
+ forceBuild = True, # use this to test your slave once it is set up
+ forceAllBuilds = False,
+ pingBuilder = False,
+ stopBuild = False,
+ stopAllBuilds = False,
+ cancelPendingBuild = False,
+)
+c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))
+
+####### PROJECT IDENTITY
+
+# the 'projectName' string will be used to describe the project that this
+# buildbot is working on. For example, it is used as the title of the
+# waterfall HTML page. The 'projectURL' string will be used to provide a link
+# from buildbot HTML pages to your project's home page.
+
+c['projectName'] = "Pyflakes"
+c['projectURL'] = "http://divmod.org/trac/wiki/DivmodPyflakes";
+
+# the 'buildbotURL' string should point to the location where the buildbot's
+# internal web server (usually the html.WebStatus page) is visible. This
+# typically uses the port number set in the Waterfall 'status' entry, but
+# with an externally-visible host name which the buildbot cannot figure out
+# without some help.
+
+c['buildbotURL'] = "http://localhost:8010/";
+
+####### DB URL
+
+# This specifies what database buildbot uses to store change and scheduler
+# state. You can leave this at its default for all but the largest
+# installations.
+c['db_url'] = "sqlite:///state.sqlite"
+
=== added file 'hooks/buildbot-relation-changed'
--- hooks/buildbot-relation-changed 1970-01-01 00:00:00 +0000
+++ hooks/buildbot-relation-changed 2012-02-06 23:20:22 +0000
@@ -0,0 +1,54 @@
+#!/usr/bin/env python
+
+import json
+import os
+
+from helpers import (
+ get_config,
+ load_pickle,
+ log,
+ relation_get,
+ relation_set,
+ save_pickle,
+ )
+from local import (
+ buildbot_reconfig,
+ generate_string,
+ slave_json,
+ )
+
+
+def get_or_create(key, prefix=''):
+ log("Retrieving {}.".format(key))
+ value = relation_get(key)
+ if not value:
+ log("Generating {}.".format(key))
+ value = generate_string(prefix)
+ log("{}: {}".format(key, value))
+ return value
+
+
+def update_slave_json(builders, name, passwd):
+ slave_info = slave_json.get()
+ slave_info[name] = (passwd, builders)
+ slave_json.set(slave_info)
+
+
+def main():
+ log("Retrieving builders.")
+ builders = filter(
+ None,
+ (b.strip() for b in relation_get('builders').split(',')))
+ log("builders: {}".format(builders))
+ name = get_or_create('name', prefix='slave-')
+ passwd = get_or_create('passwd')
+ update_slave_json(builders, name, passwd)
+ log("Reconfiguring buildbot.")
+ buildbot_reconfig()
+ log("Sending name and password to the slave.")
+ relation_set(name=name, passwd=passwd)
+
+
+if __name__ == '__main__':
+ log('BUILDBOT-RELATION-CHANGED HOOK:')
+ main()
=== modified file 'hooks/config-changed'
--- hooks/config-changed 2012-02-03 14:21:51 +0000
+++ hooks/config-changed 2012-02-06 23:20:22 +0000
@@ -1,148 +1,168 @@
#!/usr/bin/python
+import base64
+import json
+import os
+import os.path
+import shutil
+import StringIO
+import sys
+
from helpers import (
+ apt_get_install,
command,
DictDiffer,
get_config,
install_extra_repository,
- load_pickle,
- save_pickle,
+ log,
run,
)
-from subprocess import CalledProcessError
-import base64
-import os
-import os.path
-import sys
-
-# config_file is changed via juju like:
-# juju set buildbot-master config-file=`uuencode master.cfg`
-
-CONFIG_PICKLE = "config.pkl"
+from local import (
+ buildbot_create,
+ buildbot_reconfig,
+ config_json,
+ generate_string,
+ slave_json,
+ )
+
REQUIRED_KEYS = [
'buildbot-pkg',
'installdir',
]
SUPPORTED_TRANSPORTS = ['bzr']
-restart_required = False
-log = command('juju-log')
bzr = command('bzr')
-apt_get_install = command('apt-get', 'install', '-y', '--force-yes')
-
-# Log the fact that we're about to begin the install step.
-log('--> config-changed')
-
-config = get_config()
-log(str(config))
-
-# If all of the required keys are not present in the configuration then exit
-# with an error.
-if not all(config.get(k) for k in REQUIRED_KEYS):
- log('All required items not configured: {}'.format(REQUIRED_KEYS))
+
+
+def check_config(config):
+ # If all of the required keys are not present in the configuration
+ # then exit with an error.
+ if not all(config.get(k) for k in REQUIRED_KEYS):
+ log('All required items not configured: {}'.format(REQUIRED_KEYS))
+ return False
+ # If asked to fetch the config file then it must be a transport we support.
+ config_transport = config.get('config-transport')
+ config_url = config.get('config-url')
+ if config_url and config_transport not in SUPPORTED_TRANSPORTS:
+ log("{} is an unsupported transport".format(config_transport))
+ return False
+ return True
+
+
+def handle_config_changes(config, diff):
+ log('Updating buildbot configuration.')
+ log('Configuration changes seen:')
+ log(str(diff))
+
+ buildbot_pkg = config.get('buildbot-pkg')
+ extra_repo = config.get('extra-repository')
+ extra_pkgs = config.get('extra-packages')
+ restart_required = False
+
+ # Add a new repository if it was just added.
+ if extra_repo and 'extra-repository' in diff.added_or_changed:
+ install_extra_repository(extra_repo)
+ restart_required = True
+ if extra_pkgs and 'extra_packages' not in diff.unchanged:
+ apt_get_install(
+ *(pkg.strip() for pkg in extra_pkgs.split()))
+ restart_required = True
+ if 'buildbot-pkg' not in diff.unchanged:
+ apt_get_install(buildbot_pkg)
+ restart_required = True
+ return restart_required
+
+
+def configure_buildbot(config, diff):
+ buildbot_dir = config.get('installdir')
+ config_file = config.get('config-file')
+ master_cfg_path = os.path.join(buildbot_dir, 'master.cfg')
+ config_transport = config.get('config-transport')
+ config_url = config.get('config-url')
+ restart_required = False
+
+ # Write the buildbot config to disk (fetching it if necessary).
+ log("CONFIG FILE: {}".format(config_file))
+ log("ADDED OR CHANGED: {}".format(diff.added_or_changed))
+ if config_file and 'config-file' in diff.added_or_changed:
+ buildbot_create(buildbot_dir)
+ # This file will be moved to master.cfg.original in
+ # initialize_buildbot().
+ with open(master_cfg_path, 'w') as f:
+ base64.decode(StringIO.StringIO(config_file), f)
+ log('config_file decoded and written.')
+ restart_required = True
+ elif (config_transport == 'bzr' and config_url and
+ 'config-transport' not in diff.unchanged and
+ 'config-url' not in diff.unchanged):
+ # If the branch is private then more setup needs to be done. The
+ # gpg-agent needs to send the key over and the bzr launchpad-login
+ # needs to be set.
+ lp_id = config.get('config-user')
+ if lp_id:
+ bzr('launchpad-login', lp_id)
+
+ private_key = config.get('config-private-key')
+ if private_key:
+ # Set up the .ssh directory.
+ ssh_dir = os.expanduser('~/.ssh')
+ os.mkdir(ssh_dir)
+ os.chmod(ssh_dir, 0700)
+ with open(os.path.join(ssh_dir, 'lp_key', w)) as f:
+ f.write(base64.decode(private_key))
+ bzr('branch', '--use-existing-dir', config_url, buildbot_dir)
+ log('configuration fetched from {}'.format(config_url))
+ restart_required = True
+ return restart_required
+
+
+def initialize_buildbot(config):
+ # Initialize the buildbot directory and (re)start buildbot.
+ buildbot_dir = config.get('installdir')
+ master_cfg_path = os.path.join(buildbot_dir, 'master.cfg')
+ shutil.move(master_cfg_path, master_cfg_path + '.original')
+ shutil.copy(
+ os.path.join(os.path.dirname(__file__), 'master.cfg'), master_cfg_path)
+ placeholder_path = os.path.join(buildbot_dir, 'placeholder.json')
+ if not os.path.exists(placeholder_path):
+ with open(placeholder_path, 'w') as f:
+ json.dump(generate_string("temporary-placeholder-"), f)
+ run('chown', '-R', 'ubuntu:ubuntu', buildbot_dir)
+ buildbot_reconfig()
+
+
+def main():
+ config = get_config()
log(str(config))
- sys.exit(1)
-
-prev_config = load_pickle(CONFIG_PICKLE)
-diff = DictDiffer(config, prev_config)
-
-if not diff.modified:
- log("No configuration changes, exiting.")
- sys.exit(0)
-
-log('Updating buildbot configuration.')
-log('Configuration changes seen:')
-log(str(diff))
-
-buildbot_pkg = config.get('buildbot-pkg')
-buildbot_dir = config.get('installdir')
-config_file = config.get('config-file')
-config_transport = config.get('config-transport')
-config_url = config.get('config-url')
-extra_repo = config.get('extra-repository')
-extra_pkgs = config.get('extra-packages')
-
-# Add a new repository if it was just added.
-if extra_repo and 'extra-repository' in diff.added_or_changed:
- install_extra_repository(extra_repo)
- restart_required = True
-
-if extra_pkgs and 'extra_packages' not in diff.unchanged:
- apt_get_install(extra_pkgs)
- restart_required = True
-
-if 'buildbot-pkg' not in diff.unchanged:
- apt_get_install(buildbot_pkg)
- restart_required = True
-
-# Ensure the install directory exists.
-if not os.path.exists(buildbot_dir):
- os.makedirs(buildbot_dir)
-
-# If asked to fetch the config file then it must be a transport we support.
-if config_url and config_transport not in SUPPORTED_TRANSPORTS:
- log("{} is an unsupported transport".format(config_transport))
- sys.exit(1)
-
-# Write the buildbot config to disk (fetching it if necessary).
-if config_file and 'config-file' not in diff.unchanged:
+ if not check_config(config):
+ log("Configuration not valid.")
+ sys.exit(1)
+ prev_config = config_json.get()
+ diff = DictDiffer(config, prev_config)
+
+ if not diff.modified:
+ log("No configuration changes, exiting.")
+ sys.exit(0)
+
+ # Ensure the install directory exists.
+ buildbot_dir = config.get('installdir')
if not os.path.exists(buildbot_dir):
os.makedirs(buildbot_dir)
- with open(os.path.join(buildbot_dir, 'master.cfg', 'w')) as f:
- f.write(base64.decode(config_file))
- log('config_file decoded and written.')
- restart_required = True
-elif (config_transport == 'bzr' and config_url and
- 'config-transport' not in diff.unchanged and
- 'config-url' not in diff.unchanged):
- # If the branch is private then more setup needs to be done. The
- # gpg-agent needs to send the key over and the bzr launchpad-login
- # needs to be set.
- lp_id = config.get('config-user')
- if lp_id:
- bzr('launchpad-login', lp_id)
-
- private_key = config.get('config-private-key')
- if private_key:
- # Set up the .ssh directory.
- ssh_dir = os.expanduser('~/.ssh')
- os.mkdir(ssh_dir)
- os.chmod(ssh_dir, 0700)
- with open(os.path.join(ssh_dir, 'lp_key', w)) as f:
- f.write(base64.decode(private_key))
- bzr('branch', '--use-existing-dir', config_url, buildbot_dir)
- run('chown', '-R', 'ubuntu:ubuntu', buildbot_dir)
- log('configuration fetched from {}'.format(config_url))
- restart_required = True
-else:
- # Configuration file specifiers are unchanged or unrecognized.
- pass
-
-# Restart buildbot if it is running.
-if restart_required:
- pidfile = os.path.join(buildbot_dir, 'twistd.pid')
- if os.path.exists(pidfile):
- buildbot_pid = open(pidfile).read().strip()
- try:
- # Is buildbot running?
- run('kill', '-0', buildbot_pid)
- except CalledProcessError:
- # Buildbot isn't running, so no need to reconfigure it.
- pass
- else:
- # Buildbot is running, reconfigure it.
- log('Reconfiguring buildbot')
- run('buildbot', 'reconfig', buildbot_dir)
- log('Buildbot reconfigured')
+
+ restart_required = (
+ handle_config_changes(config, diff) or
+ configure_buildbot(config, diff))
+
+ master_cfg_path = os.path.join(buildbot_dir, 'master.cfg')
+ if restart_required and os.path.exists(master_cfg_path):
+ initialize_buildbot(config)
else:
- # Buildbot isn't running so start afresh but only if properly
- # configured.
- if os.path.exists(os.path.join(buildbot_dir, 'master.cfg')):
- run('buildbot', 'start', buildbot_dir)
-else:
- log("Configuration changed but didn't require restarting.")
-
-save_pickle(config, CONFIG_PICKLE)
-
-log('<-- config-changed')
+ log("Configuration changed but didn't require restarting.")
+
+ config_json.set(config)
+
+
+if __name__ == '__main__':
+ log('CONFIG-CHANGED HOOK:')
+ main()
=== modified file 'hooks/helpers.py'
--- hooks/helpers.py 2012-02-06 14:54:04 +0000
+++ hooks/helpers.py 2012-02-06 23:20:22 +0000
@@ -5,13 +5,19 @@
__metaclass__ = type
__all__ = [
+ 'apt_get_install',
'command',
'DictDiffer',
+ 'generate_string',
'get_config',
+ 'get_value_from_line',
+ 'grep',
'install_extra_repository',
'load_pickle',
'log',
'run',
+ 'relation_get',
+ 'relation_set',
'save_pickle',
'unit_info',
]
@@ -51,6 +57,7 @@
log = command('juju-log')
+apt_get_install = command('apt-get', 'install', '-y', '--force-yes')
def run(*args):
@@ -81,11 +88,42 @@
return prev_config
+def relation_get(*args):
+ cmd = command('relation-get')
+ return cmd(*args).strip()
+
+
+def relation_set(**kwargs):
+ cmd = command('relation-set')
+ args = ['{}={}'.format(k, v) for k, v in kwargs.items()]
+ return cmd(*args)
+
+
def save_pickle(obj, filepath):
with open(filepath, 'w') as fd:
pickle.dump(obj, fd)
+def unit_info(service_name, item_name, data=None):
+ if data is None:
+ data = yaml.safe_load(run('juju', 'status'))
+ services = data['services'][service_name]
+ units = services['units']
+ item = units.items()[0][1][item_name]
+ return item
+
+
+def grep(content, filename):
+ with open(filename) as f:
+ for line in f:
+ if re.match(content, line):
+ return line.strip()
+
+
+def get_value_from_line(line):
+ return line.split('=')[1].strip('"\' ')
+
+
class DictDiffer:
"""
Calculate the difference between two dictionaries as:
@@ -152,10 +190,24 @@
return s
-def unit_info(service_name, item_name, data=None):
- if data is None:
- data = yaml.safe_load(run('juju', 'status'))
- services = data['services'][service_name]
- units = services['units']
- item = units.items()[0][1][item_name]
- return item
+class Serializer:
+
+ def __init__(self, path, default=None, serialize=None, deserialize=None):
+ self.path = path
+ log("Serializer path: " + self.path)
+ self.default = default or {}
+ self.serialize = serialize or json.dump
+ self.deserialize = deserialize or json.load
+
+ def exists(self):
+ return os.path.exists(self.path)
+
+ def get(self):
+ if self.exists():
+ with open(self.path) as f:
+ return self.deserialize(f)
+ return self.default
+
+ def set(self, data):
+ with open(self.path, 'w') as f:
+ self.serialize(data, f)
=== modified file 'hooks/install'
--- hooks/install 2012-02-03 14:21:51 +0000
+++ hooks/install 2012-02-06 23:20:22 +0000
@@ -1,34 +1,49 @@
#!/usr/bin/python
-from helpers import command, get_config, run, log
-from subprocess import CalledProcessError
import os
import shutil
-
-log = command('juju-log')
-
-# Log the fact that we're about to begin the install step.
-log('--> install')
+from subprocess import CalledProcessError
+
+from helpers import (
+ apt_get_install,
+ get_config,
+ log,
+ run,
+ )
+from local import (
+ config_json,
+ slave_json,
+ )
config = get_config()
-log("config:")
-log(str(config))
-
-buildbot_dir = config['installdir']
-run('apt-get', 'install', '-y', 'sharutils', 'bzr')
-
-# Install the extra repository
-# Install the initially configured version of buildbot.
-
-# Since we may be installing into a pre-existing service, ensure the
-# buildbot directory is removed.
-if os.path.exists(buildbot_dir):
- try:
- run('buildbot', 'stop', buildbot_dir)
- except CalledProcessError:
- # It probably wasn't running; just ignore the error.
- pass
- shutil.rmtree(buildbot_dir)
-
-# Log the fact that the install step is done.
-log('<-- install')
+
+
+def bootstrap(buildbot_dir):
+ apt_get_install('sharutils', 'bzr')
+ # Since we may be installing into a pre-existing service, ensure the
+ # buildbot directory is removed.
+ if os.path.exists(buildbot_dir):
+ try:
+ run('buildbot', 'stop', buildbot_dir)
+ except (CalledProcessError, OSError):
+ # This usually happens because buildbot hasn't been
+ # installed yet, or that it wasn't running; just ignore the
+ # error.
+ pass
+ shutil.rmtree(buildbot_dir)
+ # Initialize the cached config so that old configs don't hang around
+ # after the service is torn down.
+ config_json.set({})
+ slave_json.set({})
+
+
+def main():
+ config = get_config()
+ log("config:")
+ log(str(config))
+ bootstrap(config['installdir'])
+
+
+if __name__ == '__main__':
+ log('INSTALL HOOK:')
+ main()
=== added file 'hooks/local.py'
--- hooks/local.py 1970-01-01 00:00:00 +0000
+++ hooks/local.py 2012-02-06 23:20:22 +0000
@@ -0,0 +1,132 @@
+# Copyright 2012 Canonical Ltd. This software is licensed under the
+# GNU Affero General Public License version 3 (see the file LICENSE).
+
+"""Shared functions for the buildbot master and slave"""
+
+__metaclass__ = type
+__all__ = [
+ 'buildbot_reconfig',
+ 'buildslave_start',
+ 'buildslave_stop',
+ 'config_json'
+ 'create_slave',
+ 'slave_json',
+ ]
+
+import os
+import subprocess
+import uuid
+
+from helpers import (
+ get_config,
+ log,
+ run,
+ Serializer,
+ )
+from subprocess import CalledProcessError
+from textwrap import dedent
+
+
+def _get_buildbot_dir():
+ config = get_config()
+ return config.get('installdir')
+
+
+def generate_string(prefix=""):
+ """Generate a unique string and return it."""
+ return prefix + uuid.uuid4().hex
+
+
+def _get_slave_info_path():
+ return os.path.join('/', 'tmp', 'slave_info.json')
+
+
+def buildbot_create(buildbot_dir):
+ """Create a buildbot instance in `buildbot_dir`."""
+ if not os.path.exists(os.path.join(buildbot_dir, 'buildbot.tac')):
+ return run('buildbot', 'create-master', buildbot_dir)
+
+
+def buildbot_running(buildbot_dir):
+ pidfile = os.path.join(buildbot_dir, 'twistd.pid')
+ if os.path.exists(pidfile):
+ buildbot_pid = open(pidfile).read().strip()
+ try:
+ # Is buildbot running?
+ run('kill', '-0', buildbot_pid)
+ except CalledProcessError:
+ return False
+ return True
+ return False
+
+
+def buildbot_stop():
+ buildbot_dir = _get_buildbot_dir()
+ if buildbot_running(buildbot_dir):
+ # Buildbot is running, stop it.
+ log('Stopping buildbot')
+ run('buildbot', 'stop', buildbot_dir)
+
+
+def buildbot_reconfig():
+ buildbot_dir = _get_buildbot_dir()
+ pidfile = os.path.join(buildbot_dir, 'twistd.pid')
+ running = False
+ if os.path.exists(pidfile):
+ buildbot_pid = open(pidfile).read().strip()
+ try:
+ # Is buildbot running?
+ run('kill', '-0', buildbot_pid)
+ except CalledProcessError:
+ # Buildbot isn't running, so no need to reconfigure it.
+ pass
+ else:
+ # Buildbot is running, reconfigure it.
+ log('--> Reconfiguring buildbot')
+ # XXX as root! :-(
+ # reconfig is broken in 0.8.3 (Oneiric)
+ # run('buildbot', 'reconfig', buildbot_dir)
+ run('buildbot', 'stop', buildbot_dir)
+ run('buildbot', 'start', buildbot_dir)
+ log('<-- Reconfiguring buildbot')
+ running = True
+ if not running:
+ # Buildbot isn't running so start afresh.
+ if os.path.exists(os.path.join(buildbot_dir, 'master.cfg')):
+ log('--> Starting buildbot')
+ # XXX as root! :-(
+ run('buildbot', 'start', buildbot_dir)
+ log('<-- Starting buildbot')
+
+
+def _get_tac_filename(buildbot_dir):
+ return os.path.join(buildbot_dir, 'buildbot.tac')
+
+
+def buildslave_stop(buildbot_dir=None):
+ if buildbot_dir is None:
+ buildbot_dir = _get_buildbot_dir()
+ exit_code = subprocess.call(['buildslave', 'stop', buildbot_dir])
+ tac_file = _get_tac_filename(buildbot_dir)
+ if os.path.exists(tac_file):
+ os.remove(tac_file)
+ return exit_code
+
+
+def buildslave_start(buildbot_dir=None):
+ if buildbot_dir is None:
+ buildbot_dir = _get_buildbot_dir()
+ return subprocess.call(['buildslave', 'start', buildbot_dir])
+
+
+def create_slave(name, passwd, host='localhost', buildbot_dir=None):
+ if buildbot_dir is None:
+ buildbot_dir = _get_buildbot_dir()
+ if not os.path.exists(buildbot_dir):
+ os.makedirs(buildbot_dir)
+ return subprocess.call([
+ 'buildslave', 'create-slave', buildbot_dir, host, name, passwd])
+
+
+slave_json = Serializer('/tmp/slave_info.json')
+config_json = Serializer('/tmp/config.json')
=== added file 'hooks/master.cfg'
--- hooks/master.cfg 1970-01-01 00:00:00 +0000
+++ hooks/master.cfg 2012-02-06 23:20:22 +0000
@@ -0,0 +1,38 @@
+# -*- python -*-
+# ex: set syntax=python:
+
+with open('master.cfg.original') as f:
+ exec f
+
+import uuid
+import json
+
+# Stability in this name keeps the Buildbot UI stable across live reconfigs.
+with open('placeholder.json') as f:
+ name_of_dummy_slave = json.load(f)
+
+# Now BuildmasterConfig is in the locals, configured as master.cfg.original
+# wanted it.
+c = BuildmasterConfig
+
+# This is a dict. The keys are the slave names. The values are a
+# pair of (password, builders). "builders" is a list.
+with open('slave_info.json') as f:
+ slave_info = json.load(f)
+
+# Add slaves.
+from buildbot.buildslave import BuildSlave
+c['slaves'].append(BuildSlave(name_of_dummy_slave, uuid.uuid4().hex))
+for name, (passwd, builders) in slave_info.items():
+ c['slaves'].append(BuildSlave(name, passwd))
+
+# Add slaves to builders.
+for builder in c['builders']:
+ name = builder.name
+ current_slaves = list(filter(None, builder.slavenames))
+ for slavename, (passwd, builders) in slave_info.items():
+ if name in builders:
+ current_slaves.append(slavename)
+ if not current_slaves:
+ current_slaves.append(name_of_dummy_slave)
+ builder.slavenames = current_slaves
=== removed file 'hooks/relation-name-relation-broken'
--- hooks/relation-name-relation-broken 2012-01-13 16:21:47 +0000
+++ hooks/relation-name-relation-broken 1970-01-01 00:00:00 +0000
@@ -1,2 +0,0 @@
-#!/bin/sh
-# This hook runs when the full relation is removed (not just a single member)
=== removed file 'hooks/relation-name-relation-changed'
--- hooks/relation-name-relation-changed 2012-01-13 16:21:47 +0000
+++ hooks/relation-name-relation-changed 1970-01-01 00:00:00 +0000
@@ -1,9 +0,0 @@
-#!/bin/bash
-# This must be renamed to the name of the relation. The goal here is to
-# affect any change needed by relationships being formed, modified, or broken
-# This script should be idempotent.
-juju-log $JUJU_REMOTE_UNIT modified its settings
-juju-log Relation settings:
-relation-get
-juju-log Relation members:
-relation-list
=== removed file 'hooks/relation-name-relation-departed'
--- hooks/relation-name-relation-departed 2012-01-13 16:21:47 +0000
+++ hooks/relation-name-relation-departed 1970-01-01 00:00:00 +0000
@@ -1,5 +0,0 @@
-#!/bin/sh
-# This must be renamed to the name of the relation. The goal here is to
-# affect any change needed by the remote unit leaving the relationship.
-# This script should be idempotent.
-juju-log $JUJU_REMOTE_UNIT departed
=== removed file 'hooks/relation-name-relation-joined'
--- hooks/relation-name-relation-joined 2012-01-13 16:21:47 +0000
+++ hooks/relation-name-relation-joined 1970-01-01 00:00:00 +0000
@@ -1,5 +0,0 @@
-#!/bin/sh
-# This must be renamed to the name of the relation. The goal here is to
-# affect any change needed by relationships being formed
-# This script should be idempotent.
-juju-log $JUJU_REMOTE_UNIT joined
=== modified file 'hooks/start'
--- hooks/start 2012-02-03 14:21:51 +0000
+++ hooks/start 2012-02-06 23:20:22 +0000
@@ -1,19 +1,13 @@
#!/usr/bin/python
-from helpers import (
- command,
- get_config,
- log,
- run,
- )
-
-# Log the fact that we're about to begin the start step.
-log('--> start')
-
-config = get_config()
-buildbot_dir = config.get('installdir')
-# Actually starting the buildbot happens in config-changed.
-run('open-port', '8010/TCP')
-
-# Log the fact that the start step is done.
-log('<-- start')
+from helpers import log, run
+
+
+def main():
+ # Actually starting the buildbot happens in config-changed.
+ run('open-port', '8010/TCP')
+
+
+if __name__ == '__main__':
+ log('START HOOK:')
+ main()
=== modified file 'hooks/stop'
--- hooks/stop 2012-01-30 16:43:59 +0000
+++ hooks/stop 2012-02-06 23:20:22 +0000
@@ -1,16 +1,16 @@
-#!/bin/bash
-# This will be run when the service is being torn down, allowing you to disable
-# it in various ways..
-# For example, if your web app uses a text file to signal to the load balancer
-# that it is live... you could remove it and sleep for a bit to allow the load
-# balancer to stop sending traffic.
-# rm /srv/webroot/server-live.txt && sleep 30
-
-BUILDBOT_DIR=`config-get installdir`
-
-juju-log "<-- stop"
-juju-log "Stopping buildbot in $BUILDBOT_DIR"
-close-port 8010/TCP
-buildbot stop $BUILDBOT_DIR
-juju-log "Finished stopping buildbot"
-juju-log "--> stop"
+#!/usr/bin/python
+
+from helpers import get_config, log, run
+
+
+def main():
+ config = get_config()
+ log('Stopping buildbot in {}'.format(config['installdir']))
+ run('close-port', '8010/TCP')
+
+ log('Finished stopping buildbot')
+
+
+if __name__ == '__main__':
+ log('STOP HOOK:')
+ main()
Follow ups
