launchpad-reviewers team mailing list archive
-
launchpad-reviewers team
-
Mailing list archive
-
Message #14776
[Merge] lp:~abentley/lp-dev-utils/testr-remote into lp:lp-dev-utils
Aaron Bentley has proposed merging lp:~abentley/lp-dev-utils/testr-remote into lp:lp-dev-utils.
Commit message:
Add testr-remote.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~abentley/lp-dev-utils/testr-remote/+merge/139249
Add testr-remote to lp-dev-utils
--
https://code.launchpad.net/~abentley/lp-dev-utils/testr-remote/+merge/139249
Your team Launchpad code reviewers is requested to review the proposed merge of lp:~abentley/lp-dev-utils/testr-remote into lp:lp-dev-utils.
=== modified file '.bzrignore'
--- .bzrignore 2012-08-09 04:39:28 +0000
+++ .bzrignore 2012-12-11 16:13:28 +0000
@@ -8,3 +8,6 @@
parts
bin
develop-eggs
+testr-remote/.testrepository
+testr-remote/servers
+testr-remote/server-locks/*
=== modified file 'cs-test'
--- cs-test 2012-11-09 16:37:50 +0000
+++ cs-test 2012-12-11 16:13:28 +0000
@@ -27,4 +27,9 @@
su ubuntu -c 'make schema'
EOT
. $HOME/.canonistack/novarc
-euca-run-instances ami-00000168 -t m1.small -k ${OS_USERNAME}_$OS_REGION_NAME --user-data-file $cloud_init_script
+if [ -z "$1" ]; then
+ server_count=1;
+else
+ server_count=$1;
+fi
+euca-run-instances -n $server_count ami-00000168 -t m1.small -k ${OS_USERNAME}_$OS_REGION_NAME --user-data-file $cloud_init_script
=== added directory 'testr-remote'
=== added file 'testr-remote/.testr.conf'
--- testr-remote/.testr.conf 1970-01-01 00:00:00 +0000
+++ testr-remote/.testr.conf 2012-12-11 16:13:28 +0000
@@ -0,0 +1,4 @@
+[DEFAULT]
+test_command=./test-remote --subunit $IDOPTION $LISTOPT
+test_id_option=--load-list $IDFILE
+test_list_option=--list-tests
=== added file 'testr-remote/README'
--- testr-remote/README 1970-01-01 00:00:00 +0000
+++ testr-remote/README 2012-12-11 16:13:28 +0000
@@ -0,0 +1,53 @@
+How to use
+==========
+
+You'll need a version of test-repository that supports the --concurrency parameter. Trunk will do.
+
+To use testr-remote, you first need to initialize the testr-remote directory as a test repository.
+
+To create your instances, use cs-test.
+
+Optionally, use instance-keys to add all the instances' keys to
+.ssh/known-hosts. This is better than simply accepting the key proposed by
+ssh, because it verifies the key against the fingerprint from the console
+output.
+
+Wait for the instances to finish installing Launchpad. This usually takes
+about half an hour, so it's good to do this before you need to run any tests.
+You can use a parallel ssh tool (e.g. cluster ssh) to tell you when they're
+done, because the load average will be near zero.
+
+Create a file called 'servers' in the tree root::
+
+ $(./list-servers) > servers
+
+This is just a list of the server names that can be used for ssh. You can edit
+this list if you like.
+
+Run the tests using "testr-remote". It's a wrapper around "testr-run" that
+provides the --parallel and --concurrency options, and takes the same
+parameters. e.g. "testr-remote -- -t test_project" will run
+"bin/test -t test_project" remotely. (Any parameters that will be passed on to
+bin/test need to be double-escaped. This is a bug in testr that I have a local
+fix for.)
+
+When you commit new changes, you can use "update-all" to update all servers to
+the newer version.
+
+You can also use switch-all to switch all servers to a new branch. (This does
+not re-run make).
+
+You can use "euca-terminate-instances $(./list-instance-ids)" to kill all the
+instances when you're done.
+
+How it works
+============
+testr requires a command that it can run to get subunit output, and it needs to
+run it multiple times simultaneously for concurrency. test-remote is that
+command. It provides subunit output by ssh-ing into the remote machine and
+running bin/test --subunit. It uses the server-locks directory to acquire a
+free server. If none are available, it blocks until one becomes available.
+
+When --parallel is specified, testr supplies a local file containing a list of
+test-ids to use. test-remote intercepts that option, copies the file to the
+remote server, and supplies the remote path to bin/test.
=== added file 'testr-remote/instance-key'
--- testr-remote/instance-key 1970-01-01 00:00:00 +0000
+++ testr-remote/instance-key 2012-12-11 16:13:28 +0000
@@ -0,0 +1,142 @@
+#!/usr/bin/env python
+
+__metaclass__ = type
+
+import os.path
+from optparse import OptionParser
+import re
+from subprocess import Popen, PIPE
+import sys
+from tempfile import NamedTemporaryFile
+import time
+
+
+class Instance:
+
+ def __init__(self, instance_id, ip, servername):
+
+ self.instance_id = instance_id
+ self.ip = ip
+ self.servername = servername
+ self.instance_fingerprint = None
+ self.host_key = None
+
+ def find_fingerprint(self):
+ proc = Popen(['euca-get-console-output', self.instance_id],
+ stdout=PIPE)
+ lines = iter(proc.stdout)
+ instance_key = None
+ for line in lines:
+ match = re.match('Generating public/private rsa key pair', line)
+ if match is not None:
+ [x for x in zip(lines, range(2))]
+ self.instance_fingerprint = lines.next()[:3*16-1]
+ proc.wait()
+
+
+def iter_instance():
+ proc = Popen(['euca-describe-instances'], stdout=PIPE)
+ try:
+ for num, line in enumerate(proc.stdout):
+ if not line.startswith('INSTANCE'):
+ continue
+ field = line.split('\t')
+ yield Instance(field[1], field[16], field[3])
+ finally:
+ proc.wait()
+
+
+def get_argmap(identifiers):
+ argmap = {}
+ for instance in iter_instance():
+ argmap[instance.instance_id] = instance
+ argmap[instance.ip] = instance
+ argmap[instance.servername] = instance
+ try:
+ return (argmap[identifier] for identifier in identifiers)
+ except KeyError as e:
+ sys.stderr.write('Cannot find %s\n' % e)
+ sys.exit(1)
+
+
+def find_host_keys(instances):
+ ips = [instance.ip for instance in instances]
+ proc = Popen(
+ ['ssh', 'chinstrap.canonical.com', 'ssh-keyscan'] + ips, stdout=PIPE,
+ stderr=PIPE)
+ keys = proc.communicate()[0]
+ key_map = {}
+ for key in keys.splitlines(True):
+ key_map[key.split(' ')[0]] = key
+ for instance in instances:
+ instance.host_key = key_map[instance.ip]
+
+
+def get_key_fingerprint(key):
+ with NamedTemporaryFile() as f:
+ f.write(key)
+ f.flush()
+ proc = Popen(['ssh-keygen', '-lf', f.name], stdout=PIPE)
+ host_fingerprint = proc.communicate()[0]
+ return host_fingerprint.split(' ')[1]
+
+
+def remove_key(known_hosts, hostname):
+ proc = Popen(['ssh-keygen', '-R', hostname, '-f', known_hosts],
+ stderr=PIPE)
+ proc.communicate()
+
+
+def update_key(instance):
+ known_hosts = os.path.join(os.environ['HOME'], '.ssh/known_hosts')
+ hostnames = [instance.ip, instance.servername,
+ instance.servername + '.canonistack']
+ for name in hostnames:
+ remove_key(known_hosts, name)
+ with file(known_hosts, 'ab') as f:
+ for name in hostnames:
+ f.write(instance.host_key.replace(instance.ip, name))
+
+def validate_instances(instances):
+ for instance in instances:
+ host_fingerprint = get_key_fingerprint(instance.host_key)
+ yield instance, bool(host_fingerprint == instance.instance_fingerprint)
+
+if __name__ == '__main__':
+ parser = OptionParser()
+ parser.add_option('--poll', action='store_true')
+ options, identifiers = parser.parse_args(sys.argv[1:])
+
+ if len(identifiers) > 0:
+ instances = list(get_argmap(identifiers))
+ else:
+ instances = list(iter_instance())
+
+ for instance in instances:
+ if options.poll:
+ attempt_count = 10
+ else:
+ attempt_count = 1
+ for attempt in range(attempt_count):
+ if attempt != 0:
+ sys.stdout.write('.')
+ sys.stdout.flush()
+ time.sleep(5)
+ instance.find_fingerprint()
+ if instance.instance_fingerprint is not None:
+ break
+ if attempt != 0:
+ sys.stdout.write('\n')
+
+ if instance.instance_fingerprint is None:
+ sys.stderr.write('Cannot find key in console output for %s\n' %
+ instance.instance_id)
+ sys.exit(1)
+
+ find_host_keys(instances)
+ for instance, valid in validate_instances(instances):
+ if valid:
+ print 'Match. Adding key for %s.' % instance.ip
+ update_key(instance)
+ else:
+ print "Mismatch. Not adding key."
=== added file 'testr-remote/list-instance-ids'
--- testr-remote/list-instance-ids 1970-01-01 00:00:00 +0000
+++ testr-remote/list-instance-ids 2012-12-11 16:13:28 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+euca-describe-instances | grep '^INSTANCE' | cut -f2
=== added file 'testr-remote/list-servers'
--- testr-remote/list-servers 1970-01-01 00:00:00 +0000
+++ testr-remote/list-servers 2012-12-11 16:13:28 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+euca-describe-instances | grep '^INSTANCE' | cut -f4
=== added directory 'testr-remote/server-locks'
=== added file 'testr-remote/switch-all'
--- testr-remote/switch-all 1970-01-01 00:00:00 +0000
+++ testr-remote/switch-all 2012-12-11 16:13:28 +0000
@@ -0,0 +1,11 @@
+#!/bin/sh
+set -e
+push_branch=$(bzr config -d $1 push_location|sed 's/bzr+ssh:\/\/bazaar.launchpad.net\//lp:/')
+revision_id=$(bzr revision-info $1|cut -f2 -d' ')
+echo $push_branch
+echo $revision_id
+servers=$(cat servers)
+for server in $servers; do
+ echo Switching $server ...
+ ssh $server bzr switch -d /opt/launchpad/launchpad $push_branch -r revid:$revision_id
+done
=== added file 'testr-remote/test-remote'
--- testr-remote/test-remote 1970-01-01 00:00:00 +0000
+++ testr-remote/test-remote 2012-12-11 16:13:28 +0000
@@ -0,0 +1,71 @@
+#!/usr/bin/env python
+__metaclass__ = type
+
+import argparse
+import errno
+import fcntl
+import os.path
+from contextlib import contextmanager
+import re
+import subprocess
+import sys
+import time
+
+def escape(iput):
+ return re.sub('[^-^a-z^A-Z^_.]', lambda m: '\\'+m.group(0), iput)
+
+
+class ServerSet:
+
+ def __init__(self, servers):
+ self.servers = servers
+
+ @classmethod
+ def from_file(cls):
+ return cls(file('servers').read().splitlines())
+
+ @contextmanager
+ def selected_server(self):
+ while True:
+ for server in self.servers:
+ filename = os.path.join('server-locks', server)
+ with file(filename, 'wb') as lockfile:
+ try:
+ fcntl.lockf(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except IOError as e:
+ if e.errno != errno.EWOULDBLOCK:
+ raise
+ else:
+ try:
+ yield server
+ finally:
+ fcntl.lockf(lockfile, fcntl.LOCK_UN)
+ return
+ else:
+ time.sleep(1)
+
+
+parser = argparse.ArgumentParser(description='Run some tests remotely.',
+ add_help=False)
+parser.add_argument('--load-list', nargs=1, type=argparse.FileType())
+known, unknown = parser.parse_known_args()
+args = [escape(arg) for arg in unknown]
+try:
+ with ServerSet.from_file().selected_server() as server:
+ command = ['ssh', server]
+ if known.load_list is not None:
+ command.extend(['id_list=$(mktemp);',
+ # 'echo $id_list;',
+ 'cat > $id_list;'])
+ args = ['--load-list', '$id_list'] + args
+ stdin = known.load_list[0]
+ else:
+ stdin = None
+ command.extend(['cd /opt/launchpad/launchpad;', 'xvfb-run',
+ 'bin/test'])
+ command.extend(args)
+ #print command
+ sys.exit(subprocess.call(command, stdin=stdin))
+except Exception as e:
+ sys.stderr.write(str(e) + '\n')
+ sys.exit(2)
=== added file 'testr-remote/testr-remote'
--- testr-remote/testr-remote 1970-01-01 00:00:00 +0000
+++ testr-remote/testr-remote 2012-12-11 16:13:28 +0000
@@ -0,0 +1,4 @@
+#!/bin/sh
+server_count=$(wc -l servers | cut -f 1 -d ' ')
+concurrency=$((server_count*2))
+~/hacking/testrepository/testr run --parallel --concurrency $concurrency "$@"
=== added file 'testr-remote/update-all'
--- testr-remote/update-all 1970-01-01 00:00:00 +0000
+++ testr-remote/update-all 2012-12-11 16:13:28 +0000
@@ -0,0 +1,2 @@
+#!/bin/sh
+for server in $(cat servers); do ssh $server bzr update /opt/launchpad/launchpad; done
