cloud-init-dev team mailing list archive
-
cloud-init-dev team
-
Mailing list archive
-
Message #01596
[Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
Wesley Wiedenmeier has proposed merging ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master.
Requested reviews:
Joshua Powers (powersj)
cloud init development team (cloud-init-dev)
For more details, see:
https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/+git/cloud-init/+merge/308218
integration-testing updates
- enable additional distros
- allow use of either pylxd 2.1.3 or 2.2
- detected at runtime during instance.execute()
- when using 2.2 the test suite cannot complete large
runs without hitting the pylxd too many open files
issue, so 2.2 should only be used for development
- when using 2.1.3, the test suite will not have proper
error handling during setup phase, as 2.1.3 does not
return command exit codes. this is adequate for use on
jenkins until the fix for 2.2 is merged in upstream
- refactor image setup code
- improve error handling throughout
- change behavior of --upgrade setup_image to only
upgrade cloud-init
- add --upgrade-all setup_image flag to upgrade all
packages on system
- clean up detection of cloud-init version
- output cloud-init version at end of setup_image
- use 'yum install' rather than 'yum update' in
setup_image.upgrade in case cloud-init not installed
- update help in setup_image args
- clean up image config
- new image config format allows finer control over
platform specific image config, with less verbosity
- clean up handling of image config throughout platform
- added additional releases and distros
- re-ordered releases.yaml in order of release date, and
added EOL comments for future reference
- allow image config to override setup options with
'setup_overrides'
- allow image config to specify preserving base images
for lxd
- allow using images from images.linuxcontainers.org for
releases other than ubuntu
- improve waiting for system to boot
- separated out determining if the system has booted from
determining if cloud-init has finished
- do not wait for cloud-init during setup phases, as it
may not be installed
- use 'systemctl is-system-running' or 'runlevel'
depending on init system
- poll for system ready inside the image rather than
through instance.execute() to make pylxd too many open
files error take longer to strike
- clean up code for handling waiting
- update lxd image metadata and templates if necessary
- many lxd images that do not come with cloud-init
installed do not have the necessary templates to write
a nocloud seed into the image
- if necessary, export image, extract metadata, add new
template scripts, and import as new image
- add new tox environments to run tests
- new versions of pylxd only available through pip on
xenial, best to avoid using pip outside a venv
- environments 'citest' and 'citest_new_pylxd' provide
entry points into the cli, allowing use of any test
commands, using pylxd 2.1.3 and 2.2.2 respectively
--
Your team cloud init development team is requested to review the proposed merge of ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master.
diff --git a/tests/cloud_tests/args.py b/tests/cloud_tests/args.py
index b68cc98..a96714c 100644
--- a/tests/cloud_tests/args.py
+++ b/tests/cloud_tests/args.py
@@ -9,11 +9,11 @@ ARG_SETS = {
'COLLECT': (
(('-p', '--platform'),
{'help': 'platform(s) to run tests on', 'metavar': 'PLATFORM',
- 'action': 'append', 'choices': config.list_enabled_platforms(),
+ 'action': 'append', 'choices': config.ENABLED_PLATFORMS,
'default': []}),
(('-n', '--os-name'),
{'help': 'the name(s) of the OS(s) to test', 'metavar': 'NAME',
- 'action': 'append', 'choices': config.list_enabled_distros(),
+ 'action': 'append', 'choices': config.ENABLED_DISTROS,
'default': []}),
(('-t', '--test-config'),
{'help': 'test config file(s) to use', 'metavar': 'FILE',
@@ -61,8 +61,12 @@ ARG_SETS = {
{'help': 'ppa to enable (implies -u)', 'metavar': 'NAME',
'action': 'store'}),
(('-u', '--upgrade'),
- {'help': 'upgrade before starting tests', 'action': 'store_true',
- 'default': False}),),
+ {'help': 'upgrade or install cloud-init from repo',
+ 'action': 'store_true', 'default': False}),
+ (('--upgrade-full',),
+ {'help': 'do full system upgrade from repo (implies -u)',
+ 'action': 'store_true', 'default': False}),),
+
}
SUBCMDS = {
@@ -121,15 +125,15 @@ def normalize_collect_args(args):
"""
# platform should default to all supported
if len(args.platform) == 0:
- args.platform = config.list_enabled_platforms()
+ args.platform = config.ENABLED_PLATFORMS
args.platform = util.sorted_unique(args.platform)
# os name should default to all enabled
# if os name is provided ensure that all provided are supported
if len(args.os_name) == 0:
- args.os_name = config.list_enabled_distros()
+ args.os_name = config.ENABLED_DISTROS
else:
- supported = config.list_enabled_distros()
+ supported = config.ENABLED_DISTROS
invalid = [os_name for os_name in args.os_name
if os_name not in supported]
if len(invalid) != 0:
diff --git a/tests/cloud_tests/collect.py b/tests/cloud_tests/collect.py
index 68b47d7..4c015cd 100644
--- a/tests/cloud_tests/collect.py
+++ b/tests/cloud_tests/collect.py
@@ -18,8 +18,10 @@ def collect_script(instance, base_dir, script, script_name):
return_value: None, may raise errors
"""
LOG.debug('running collect script: %s', script_name)
- util.write_file(os.path.join(base_dir, script_name),
- instance.run_script(script))
+ (out, err, exit) = instance.run_script(
+ script, ignore_errors=True,
+ description='collect: {}'.format(script_name))
+ util.write_file(os.path.join(base_dir, script_name), out)
def collect_test_data(args, snapshot, os_name, test_name):
@@ -39,9 +41,6 @@ def collect_test_data(args, snapshot, os_name, test_name):
test_scripts = test_config['collect_scripts']
test_output_dir = os.sep.join(
(args.data_dir, snapshot.platform_name, os_name, test_name))
- boot_timeout = (test_config.get('boot_timeout')
- if isinstance(test_config.get('boot_timeout'), int) else
- snapshot.config.get('timeout'))
# if test is not enabled, skip and return 0 failures
if not test_config.get('enabled', False):
@@ -56,7 +55,7 @@ def collect_test_data(args, snapshot, os_name, test_name):
LOG.info('collecting test data for test: %s', test_name)
with component as instance:
start_call = partial(run_single, 'boot instance', partial(
- instance.start, wait=True, wait_time=boot_timeout))
+ instance.start, wait=True, wait_for_cloud_init=True))
collect_calls = [partial(run_single, 'script {}'.format(script_name),
partial(collect_script, instance,
test_output_dir, script, script_name))
@@ -100,10 +99,8 @@ def collect_image(args, platform, os_name):
"""
res = ({}, 1)
- os_config = config.load_os_config(os_name)
- if not os_config.get('enabled'):
- raise ValueError('OS {} not enabled'.format(os_name))
-
+ os_config = config.load_os_config(
+ platform.platform_name, os_name, require_enabled=True)
component = PlatformComponent(
partial(images.get_image, platform, os_config))
@@ -126,10 +123,8 @@ def collect_platform(args, platform_name):
"""
res = ({}, 1)
- platform_config = config.load_platform_config(platform_name)
- if not platform_config.get('enabled'):
- raise ValueError('Platform {} not enabled'.format(platform_name))
-
+ platform_config = config.load_platform_config(
+ platform_name, require_enabled=True)
component = PlatformComponent(
partial(platforms.get_platform, platform_name, platform_config))
diff --git a/tests/cloud_tests/config.py b/tests/cloud_tests/config.py
index f3a13c9..01cddf7 100644
--- a/tests/cloud_tests/config.py
+++ b/tests/cloud_tests/config.py
@@ -14,6 +14,20 @@ RELEASES_CONF = os.path.join(BASE_DIR, 'releases.yaml')
TESTCASE_CONF = os.path.join(BASE_DIR, 'testcases.yaml')
+def get(base, key):
+ """
+ get config entry 'key' from base, ensuring is dictionary
+ """
+ return base[key] if key in base and base[key] is not None else {}
+
+
+def enabled(config):
+ """
+ test if config item is enabled
+ """
+ return isinstance(config, dict) and config.get('enabled', False)
+
+
def path_to_name(path):
"""
convert abs or rel path to test config to path under configs/
@@ -61,22 +75,39 @@ def merge_config(base, override):
return res
-def load_platform_config(platform):
+def load_platform_config(platform_name, require_enabled=False):
"""
load configuration for platform
+ platform_name: name of platform to retrieve config for
+ require_enabled: if true, raise error if 'enabled' not True
+ return_value: config dict
"""
main_conf = c_util.read_conf(PLATFORM_CONF)
- return merge_config(main_conf.get('default_platform_config'),
- main_conf.get('platforms')[platform])
+ conf = merge_config(main_conf['default_platform_config'],
+ main_conf['platforms'][platform_name])
+ if require_enabled and not enabled(conf):
+ raise ValueError('Platform is not enabled')
+ return conf
-def load_os_config(os_name):
+def load_os_config(platform_name, os_name, require_enabled=False):
"""
load configuration for os
+ platform_name: platform name to load os config for
+ os_name: name of os to retrieve config for
+ require_enabled: if true, raise error if 'enabled' not True
+ return_value: config dict
"""
main_conf = c_util.read_conf(RELEASES_CONF)
- return merge_config(main_conf.get('default_release_config'),
- main_conf.get('releases')[os_name])
+ default = main_conf['default_release_config']
+ image = main_conf['releases'][os_name]
+ conf = merge_config(merge_config(get(default, 'default'),
+ get(default, platform_name)),
+ merge_config(get(image, 'default'),
+ get(image, platform_name)))
+ if require_enabled and not enabled(conf):
+ raise ValueError('OS is not enabled')
+ return conf
def load_test_config(path):
@@ -91,16 +122,22 @@ def list_enabled_platforms():
"""
list all platforms enabled for testing
"""
- platforms = c_util.read_conf(PLATFORM_CONF).get('platforms')
- return [k for k, v in platforms.items() if v.get('enabled')]
+ platforms = get(c_util.read_conf(PLATFORM_CONF), 'platforms')
+ return [k for k, v in platforms.items() if enabled(v)]
-def list_enabled_distros():
+def list_enabled_distros(platforms):
"""
- list all distros enabled for testing
+ list all distros enabled for testing on specified platforms
"""
- releases = c_util.read_conf(RELEASES_CONF).get('releases')
- return [k for k, v in releases.items() if v.get('enabled')]
+
+ def platform_has_enabled(config):
+ return any(enabled(merge_config(get(config, 'default'),
+ get(config, platform)))
+ for platform in platforms)
+
+ releases = get(c_util.read_conf(RELEASES_CONF), 'releases')
+ return [k for k, v in releases.items() if platform_has_enabled(v)]
def list_test_configs():
@@ -110,4 +147,8 @@ def list_test_configs():
return [os.path.abspath(f) for f in
glob.glob(os.sep.join((TEST_CONF_DIR, '*', '*.yaml')))]
+
+ENABLED_PLATFORMS = list_enabled_platforms()
+ENABLED_DISTROS = list_enabled_distros(ENABLED_PLATFORMS)
+
# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/images/base.py b/tests/cloud_tests/images/base.py
index 394b11f..e61a928 100644
--- a/tests/cloud_tests/images/base.py
+++ b/tests/cloud_tests/images/base.py
@@ -7,13 +7,14 @@ class Image(object):
"""
platform_name = None
- def __init__(self, name, config, platform):
+ def __init__(self, platform, config):
"""
- setup
+ Set up image
+ platform: platform object
+ config: image configuration
"""
- self.name = name
- self.config = config
self.platform = platform
+ self.config = config
def __str__(self):
"""
@@ -28,10 +29,16 @@ class Image(object):
"""
raise NotImplementedError
- # FIXME: instead of having execute and push_file and other instance methods
- # here which pass through to a hidden instance, it might be better
- # to expose an instance that the image can be modified through
- def execute(self, command, stdin=None, stdout=None, stderr=None, env={}):
+ @property
+ def setup_overrides(self):
+ """
+ setup options that need to be overridden for the image
+ return_value: dictionary to update args with
+ """
+ # NOTE: more sophisticated options may be requied at some point
+ return self.config.get('setup_overrides', {})
+
+ def execute(self, *args, **kwargs):
"""
execute command in image, modifying image
"""
@@ -43,7 +50,7 @@ class Image(object):
"""
raise NotImplementedError
- def run_script(self, script):
+ def run_script(self, *args, **kwargs):
"""
run script in image, modifying image
return_value: script output
diff --git a/tests/cloud_tests/images/lxd.py b/tests/cloud_tests/images/lxd.py
index 7a41614..1fdb91e 100644
--- a/tests/cloud_tests/images/lxd.py
+++ b/tests/cloud_tests/images/lxd.py
@@ -2,6 +2,10 @@
from tests.cloud_tests.images import base
from tests.cloud_tests.snapshots import lxd as lxd_snapshot
+from tests.cloud_tests import util
+
+import os
+import shutil
class LXDImage(base.Image):
@@ -10,27 +14,43 @@ class LXDImage(base.Image):
"""
platform_name = "lxd"
- def __init__(self, name, config, platform, pylxd_image):
+ def __init__(self, platform, config, pylxd_image):
"""
- setup
+ Set up image
+ platform: platform object
+ config: image configuration
"""
- self.platform = platform
- self._pylxd_image = pylxd_image
+ self.modified = False
self._instance = None
- super(LXDImage, self).__init__(name, config, platform)
+ self._pylxd_image = None
+ self.pylxd_image = pylxd_image
+ super(LXDImage, self).__init__(platform, config)
@property
def pylxd_image(self):
- self._pylxd_image.sync()
+ if self._pylxd_image:
+ self._pylxd_image.sync()
return self._pylxd_image
+ @pylxd_image.setter
+ def pylxd_image(self, pylxd_image):
+ if self._instance:
+ self._instance.destroy()
+ self._instance = None
+ if (self._pylxd_image and
+ (self._pylxd_image is not pylxd_image) and
+ (not self.config.get('cache_base_image') or self.modified)):
+ self._pylxd_image.delete(wait=True)
+ self.modified = False
+ self._pylxd_image = pylxd_image
+
@property
def instance(self):
if not self._instance:
self._instance = self.platform.launch_container(
- image=self.pylxd_image.fingerprint,
- image_desc=str(self), use_desc='image-modification')
- self._instance.start(wait=True, wait_time=self.config.get('timeout'))
+ self.properties, self.config, use_desc='image-modification',
+ image_desc=str(self), image=self.pylxd_image.fingerprint)
+ self._instance.start()
return self._instance
@property
@@ -46,6 +66,78 @@ class LXDImage(base.Image):
'release': properties.get('release'),
}
+ def export_image(self, output_dir):
+ """
+ export image from lxd image store to (split) tarball on disk
+ output_dir: dir to store tarballs in
+ return_value: tuple of path to metadata tarball and rootfs tarball
+ """
+ # pylxd's image export feature doesn't do split exports, so use cmdline
+ util.subp(['lxc', 'image', 'export', self.pylxd_image.fingerprint,
+ output_dir], capture=True)
+ tarballs = [p for p in os.listdir(output_dir) if p.endswith('tar.xz')]
+ metadata = os.path.join(
+ output_dir, next(p for p in tarballs if p.startswith('meta-')))
+ rootfs = os.path.join(
+ output_dir, next(p for p in tarballs if not p.startswith('meta-')))
+ return (metadata, rootfs)
+
+ def import_image(self, metadata, rootfs):
+ """
+ import image to lxd image store from (split) tarball on disk
+ note, this will replace and delete the current pylxd_image
+ metadata: metadata tarball
+ rootfs: rootfs tarball
+ return_value: imported image fingerprint
+ """
+ alias = util.gen_instance_name(
+ image_desc=str(self), use_desc='update-metadata')
+ util.subp(['lxc', 'image', 'import', metadata, rootfs,
+ '--alias', alias], capture=True)
+ self.pylxd_image = self.platform.query_image_by_alias(alias)
+ return self.pylxd_image.fingerprint
+
+ def update_templates(self, template_config, template_data):
+ """
+ update the image's template configuration
+ note, this will replace and delete the current pylxd_image
+ template_config: config overrides for template portion of metadata
+ template_data: template data to place into templates/
+ """
+ # set up tmp files
+ export_dir = util.tmpdir()
+ extract_dir = util.tmpdir()
+ new_metadata = os.path.join(export_dir, 'new-meta.tar.xz')
+ metadata_yaml = os.path.join(extract_dir, 'metadata.yaml')
+ template_dir = os.path.join(extract_dir, 'templates')
+
+ try:
+ # extract old data
+ (metadata, rootfs) = self.export_image(export_dir)
+ shutil.unpack_archive(metadata, extract_dir)
+
+ # update metadata
+ metadata = util.read_conf(metadata_yaml)
+ templates = metadata.get('templates', {})
+ templates.update(template_config)
+ metadata['templates'] = templates
+ util.yaml_dump(metadata, metadata_yaml)
+
+ # write out template files
+ for name, content in template_data.items():
+ path = os.path.join(template_dir, name)
+ util.write_file(path, content)
+
+ # store new data, mark new image as modified
+ util.flat_tar(new_metadata, extract_dir)
+ self.import_image(new_metadata, rootfs)
+ self.modified = True
+
+ finally:
+ # remove tmpfiles
+ shutil.rmtree(export_dir)
+ shutil.rmtree(extract_dir)
+
def execute(self, *args, **kwargs):
"""
execute command in image, modifying image
@@ -58,12 +150,12 @@ class LXDImage(base.Image):
"""
return self.instance.push_file(local_path, remote_path)
- def run_script(self, script):
+ def run_script(self, *args, **kwargs):
"""
run script in image, modifying image
return_value: script output
"""
- return self.instance.run_script(script)
+ return self.instance.run_script(*args, **kwargs)
def snapshot(self):
"""
@@ -71,22 +163,20 @@ class LXDImage(base.Image):
"""
# clone current instance, start and freeze clone
instance = self.platform.launch_container(
- container=self.instance.name, image_desc=str(self),
- use_desc='snapshot')
- instance.start(wait=True, wait_time=self.config.get('timeout'))
+ self.properties, self.config, container=self.instance.name,
+ image_desc=str(self), use_desc='snapshot')
+ instance.start()
if self.config.get('boot_clean_script'):
instance.run_script(self.config.get('boot_clean_script'))
instance.freeze()
return lxd_snapshot.LXDSnapshot(
- self.properties, self.config, self.platform, instance)
+ self.platform, self.properties, self.config, instance)
def destroy(self):
"""
clean up data associated with image
"""
- if self._instance:
- self._instance.destroy()
- self.pylxd_image.delete(wait=True)
+ self.pylxd_image = None
super(LXDImage, self).destroy()
# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/base.py b/tests/cloud_tests/instances/base.py
index 9559d28..2711860 100644
--- a/tests/cloud_tests/instances/base.py
+++ b/tests/cloud_tests/instances/base.py
@@ -1,8 +1,5 @@
# This file is part of cloud-init. See LICENSE file for license information.
-import os
-import uuid
-
class Instance(object):
"""
@@ -10,26 +7,37 @@ class Instance(object):
"""
platform_name = None
- def __init__(self, name):
+ def __init__(self, platform, name, properties, config):
"""
- setup
+ Set up instance
+ platform: platform object
+ name: hostname of instance
+ properties: image properties
+ config: image config
"""
+ self.platform = platform
self.name = name
+ self.properties = properties
+ self.config = config
- def execute(self, command, stdin=None, stdout=None, stderr=None, env={}):
+ def execute(self, command, stdout=None, stderr=None, env={},
+ ignore_errors=False, description=None):
"""
+ Execute command in instance, recording output, error and exit code.
+ Assumes functional networking and execution as root with the
+ target filesystem being available at /.
+
command: the command to execute as root inside the image
- stdin, stderr, stdout: file handles
+ stdout, stderr: file handles to write output and error to
env: environment variables
-
- Execute assumes functional networking and execution as root with the
- target filesystem being available at /.
+ ignore_errors: do not raise an error if the command fails
+ description: purpose of command
return_value: tuple containing stdout data, stderr data, exit code
"""
raise NotImplementedError
- def read_data(self, remote_path, encode=False):
+ def read_data(self, remote_path, decode=False):
"""
read_data from instance filesystem
remote_path: path in instance
@@ -49,6 +57,8 @@ class Instance(object):
def pull_file(self, remote_path, local_path):
"""
copy file at 'remote_path', from instance to 'local_path'
+ remote_path: path on remote instance
+ local_path: path on local instance
"""
with open(local_path, 'wb') as fp:
fp.write(self.read_data(remote_path), encode=True)
@@ -56,18 +66,35 @@ class Instance(object):
def push_file(self, local_path, remote_path):
"""
copy file at 'local_path' to instance at 'remote_path'
+ local_path: path on local instance
+ remote_path: path on remote instance
"""
with open(local_path, 'rb') as fp:
self.write_data(remote_path, fp.read())
- def run_script(self, script):
+ def run_script(self, script, ignore_errors=False, description=None):
"""
run script in target and return stdout
+ script: script contents
+ ignore_errors: do not raise an error if the script returns non-zero
+ description: purpose of script
+ return_value: stdout from script
"""
- script_path = os.path.join('/tmp', str(uuid.uuid1()))
- self.write_data(script_path, script)
- (out, err, exit_code) = self.execute(['/bin/bash', script_path])
- return out
+ script_path = self.tmpfile()
+ try:
+ self.write_data(script_path, script)
+ return self.execute(['/bin/bash', script_path],
+ ignore_errors=ignore_errors,
+ description=description)
+ finally:
+ self.execute(['rm', script_path], ignore_errors=ignore_errors)
+
+ def tmpfile(self):
+ """
+ get a tmp file in the target
+ return_value: path to new file in target
+ """
+ return self.execute(['mktemp'])[0].strip()
def console_log(self):
"""
@@ -87,7 +114,7 @@ class Instance(object):
"""
raise NotImplementedError
- def start(self, wait=True):
+ def start(self, wait=True, wait_for_cloud_init=False):
"""
start instance
"""
@@ -99,22 +126,32 @@ class Instance(object):
"""
pass
- def _wait_for_cloud_init(self, wait_time):
+ def _wait_for_system(self, wait_for_cloud_init):
"""
wait until system has fully booted and cloud-init has finished
+ wait_time: maximum time to wait
+ return_value: None, may raise OSError if wait_time exceeded
"""
- if not wait_time:
- return
- found_msg = 'found'
- cmd = ('for ((i=0;i<{wait};i++)); do [ -f "{file}" ] && '
- '{{ echo "{msg}";break; }} || sleep 1; done').format(
- file='/run/cloud-init/result.json',
- wait=wait_time, msg=found_msg)
+ def clean_test(test):
+ """
+ clean formatting for system ready test testcase
+ """
+ return ' '.join(l for l in test.strip().splitlines()
+ if not l.lstrip().startswith('#'))
+
+ time = self.config['boot_timeout']
+ tests = [self.config['system_ready_script']]
+ if wait_for_cloud_init:
+ tests.append(self.config['cloud_init_ready_script'])
+
+ formatted_tests = ' && '.join(clean_test(t) for t in tests)
+ test_cmd = ('for ((i=0;i<{time};i++)); do {test} && exit 0; sleep 1; '
+ 'done; exit 1;').format(time=time, test=formatted_tests)
+ cmd = ['/bin/bash', '-c', test_cmd]
+
+ if self.execute(cmd, ignore_errors=True)[-1] != 0:
+ raise OSError('timeout: after {}s system not started'.format(time))
- (out, err, exit) = self.execute(['/bin/bash', '-c', cmd])
- if out.strip() != found_msg:
- raise OSError('timeout: after {}s, cloud-init has not started'
- .format(wait_time))
# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/instances/lxd.py b/tests/cloud_tests/instances/lxd.py
index f0aa121..72ae2b8 100644
--- a/tests/cloud_tests/instances/lxd.py
+++ b/tests/cloud_tests/instances/lxd.py
@@ -1,6 +1,7 @@
# This file is part of cloud-init. See LICENSE file for license information.
from tests.cloud_tests.instances import base
+from tests.cloud_tests import util
class LXDInstance(base.Instance):
@@ -9,41 +10,66 @@ class LXDInstance(base.Instance):
"""
platform_name = "lxd"
- def __init__(self, name, platform, pylxd_container):
+ def __init__(self, platform, name, properties, config, pylxd_container):
"""
- setup
+ Set up instance
+ platform: platform object
+ name: hostname of instance
+ properties: image properties
+ config: image config
"""
- self.platform = platform
self._pylxd_container = pylxd_container
- super(LXDInstance, self).__init__(name)
+ super(LXDInstance, self).__init__(platform, name, properties, config)
@property
def pylxd_container(self):
self._pylxd_container.sync()
return self._pylxd_container
- def execute(self, command, stdin=None, stdout=None, stderr=None, env={}):
+ def execute(self, command, stdout=None, stderr=None, env={},
+ ignore_errors=False, description=None):
"""
+ Execute command in instance, recording output, error and exit code.
+ Assumes functional networking and execution as root with the
+ target filesystem being available at /.
+
command: the command to execute as root inside the image
- stdin, stderr, stdout: file handles
+ stdout, stderr: file handles to write output and error to
env: environment variables
-
- Execute assumes functional networking and execution as root with the
- target filesystem being available at /.
+ ignore_errors: do not raise an error if the command fails
+ description: purpose of command
return_value: tuple containing stdout data, stderr data, exit code
"""
- # TODO: the pylxd api handler for container.execute needs to be
- # extended to properly pass in stdin
- # TODO: the pylxd api handler for container.execute needs to be
- # extended to get the return code, for now just use 0
+ # ensure instance is running and execute the command
self.start()
- if stdin:
- raise NotImplementedError
res = self.pylxd_container.execute(command, environment=env)
- for (f, data) in (i for i in zip((stdout, stderr), res) if i[0]):
- f.write(data)
- return res + (0,)
+
+ # get out, exit and err from pylxd return
+ if hasattr(res, 'exit_code'):
+ # pylxd 2.2 returns ContainerExecuteResult, named tuple of
+ # (exit_code, out, err)
+ (exit, out, err) = res
+ else:
+ # pylxd 2.1.3 and earlier only return out and err, no exit
+ # LOG.warning('using pylxd version < 2.2')
+ (out, err) = res
+ exit = 0
+
+ # write data to file descriptors if needed
+ if stdout:
+ stdout.write(out)
+ if stderr:
+ stderr.write(err)
+
+ # if the command failed and ignore_errors is not set, raise an error
+ if exit != 0 and not ignore_errors:
+ error_desc = ('Failed command to: {}'.format(description)
+ if description else None)
+ raise util.InTargetExecuteError(
+ out, err, exit, command, self.name, error_desc)
+
+ return (out, err, exit)
def read_data(self, remote_path, decode=False):
"""
@@ -83,14 +109,14 @@ class LXDInstance(base.Instance):
if self.pylxd_container.status != 'Stopped':
self.pylxd_container.stop(wait=wait)
- def start(self, wait=True, wait_time=None):
+ def start(self, wait=True, wait_for_cloud_init=False):
"""
start instance
"""
if self.pylxd_container.status != 'Running':
self.pylxd_container.start(wait=wait)
- if wait and isinstance(wait_time, int):
- self._wait_for_cloud_init(wait_time)
+ if wait:
+ self._wait_for_system(wait_for_cloud_init)
def freeze(self):
"""
diff --git a/tests/cloud_tests/platforms.yaml b/tests/cloud_tests/platforms.yaml
index 5972b32..b91834a 100644
--- a/tests/cloud_tests/platforms.yaml
+++ b/tests/cloud_tests/platforms.yaml
@@ -10,7 +10,55 @@ default_platform_config:
platforms:
lxd:
enabled: true
- get_image_timeout: 600
+ # overrides for image templates
+ template_overrides:
+ /var/lib/cloud/seed/nocloud-net/meta-data:
+ when:
+ - create
+ - copy
+ template: cloud-init-meta.tpl
+ /var/lib/cloud/seed/nocloud-net/network-config:
+ when:
+ - create
+ - copy
+ template: cloud-init-network.tpl
+ /var/lib/cloud/seed/nocloud-net/user-data:
+ when:
+ - create
+ - copy
+ template: cloud-init-user.tpl
+ properties:
+ default: |
+ #cloud-config
+ {}
+ /var/lib/cloud/seed/nocloud-net/vendor-data:
+ when:
+ - create
+ - copy
+ template: cloud-init-vendor.tpl
+ properties:
+ default: |
+ #cloud-config
+ {}
+ # overrides image template files
+ template_files:
+ cloud-init-meta.tpl: |
+ #cloud-config
+ instance-id: {{ container.name }}
+ local-hostname: {{ container.name }}
+ {{ config_get("user.meta-data", "") }}
+ cloud-init-network.tpl: |
+ {% if config_get("user.network-config", "") == "" %}version: 1
+ config:
+ - type: physical
+ name: eth0
+ subnets:
+ - type: {% if config_get("user.network_mode", "") == "link-local" %}manual{% else %}dhcp{% endif %}
+ control: auto{% else %}{{ config_get("user.network-config", "") }}{% endif %}
+ cloud-init-user.tpl: |
+ {{ config_get("user.user-data", properties.default) }}
+ cloud-init-vendor.tpl: |
+ {{ config_get("user.vendor-data", properties.default) }}
ec2: {}
azure: {}
diff --git a/tests/cloud_tests/platforms/base.py b/tests/cloud_tests/platforms/base.py
index 615e2e0..2b6e514 100644
--- a/tests/cloud_tests/platforms/base.py
+++ b/tests/cloud_tests/platforms/base.py
@@ -15,17 +15,7 @@ class Platform(object):
def get_image(self, img_conf):
"""
- Get image using 'img_conf', where img_conf is a dict containing all
- image configuration parameters
-
- in this dict there must be a 'platform_ident' key containing
- configuration for identifying each image on a per platform basis
-
- see implementations for get_image() for details about the contents
- of the platform's config entry
-
- note: see 'releases' main_config.yaml for example entries
-
+ get image using specified image configuration
img_conf: configuration for image
return_value: cloud_tests.images instance
"""
@@ -37,17 +27,4 @@ class Platform(object):
"""
pass
- def _extract_img_platform_config(self, img_conf):
- """
- extract platform configuration for current platform from img_conf
- """
- platform_ident = img_conf.get('platform_ident')
- if not platform_ident:
- raise ValueError('invalid img_conf, missing \'platform_ident\'')
- ident = platform_ident.get(self.platform_name)
- if not ident:
- raise ValueError('img_conf: {} missing config for platform {}'
- .format(img_conf, self.platform_name))
- return ident
-
# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/platforms/lxd.py b/tests/cloud_tests/platforms/lxd.py
index 847cc54..82e2710 100644
--- a/tests/cloud_tests/platforms/lxd.py
+++ b/tests/cloud_tests/platforms/lxd.py
@@ -27,28 +27,30 @@ class LXDPlatform(base.Platform):
def get_image(self, img_conf):
"""
- Get image
- img_conf: dict containing config for image. platform_ident must have:
- alias: alias to use for simplestreams server
- sstreams_server: simplestreams server to use, or None for default
+ get image using specified image configuration
+ img_conf: configuration for image
return_value: cloud_tests.images instance
"""
- lxd_conf = self._extract_img_platform_config(img_conf)
- image = self.client.images.create_from_simplestreams(
- lxd_conf.get('sstreams_server', DEFAULT_SSTREAMS_SERVER),
- lxd_conf['alias'])
- return lxd_image.LXDImage(
- image.properties['description'], img_conf, self, image)
+ pylxd_image = self.client.images.create_from_simplestreams(
+ img_conf.get('sstreams_server', DEFAULT_SSTREAMS_SERVER),
+ img_conf['alias'])
+ image = lxd_image.LXDImage(self, img_conf, pylxd_image)
+ if img_conf.get('override_templates', False):
+ image.update_templates(self.config.get('template_overrides', {}),
+ self.config.get('template_files', {}))
+ return image
- def launch_container(self, image=None, container=None, ephemeral=False,
- config=None, block=True,
+ def launch_container(self, properties, config, image=None, container=None,
+ ephemeral=False, container_config=None, block=True,
image_desc=None, use_desc=None):
"""
launch a container
+ properties: image properties
+ config: image configuration
image: image fingerprint to launch from
container: container to copy
ephemeral: delete image after first shutdown
- config: config options for instance as dict
+ container_config: config options for instance as dict
block: wait until container created
image_desc: description of image being launched
use_desc: description of container's use
@@ -61,11 +63,13 @@ class LXDPlatform(base.Platform):
use_desc=use_desc,
used_list=self.list_containers()),
'ephemeral': bool(ephemeral),
- 'config': config if isinstance(config, dict) else {},
+ 'config': (container_config
+ if isinstance(container_config, dict) else {}),
'source': ({'type': 'image', 'fingerprint': image} if image else
{'type': 'copy', 'source': container})
}, wait=block)
- return lxd_instance.LXDInstance(container.name, self, container)
+ return lxd_instance.LXDInstance(self, container.name, properties,
+ config, container)
def container_exists(self, container_name):
"""
@@ -88,6 +92,14 @@ class LXDPlatform(base.Platform):
"""
return [container.name for container in self.client.containers.all()]
+ def query_image_by_alias(self, alias):
+ """
+ get image by alias in local image store
+ alias: alias of image
+ return_value: pylxd image (not cloud_tests.images instance)
+ """
+ return self.client.images.get_by_alias(alias)
+
def destroy(self):
"""
Clean up platform data
diff --git a/tests/cloud_tests/releases.yaml b/tests/cloud_tests/releases.yaml
index 3ffa68f..b18b095 100644
--- a/tests/cloud_tests/releases.yaml
+++ b/tests/cloud_tests/releases.yaml
@@ -1,79 +1,145 @@
# ============================= Release Config ================================
default_release_config:
- # all are disabled by default
- enabled: false
- # timeout for booting image and running cloud init
- timeout: 120
- # platform_ident values for the image, with data to identify the image
- # on that platform. see platforms.base for more information
- platform_ident: {}
- # a script to run after a boot that is used to modify an image, before
- # making a snapshot of the image. may be useful for removing data left
- # behind from cloud-init booting, such as logs, to ensure that data from
- # snapshot.launch() will not include a cloud-init.log from a boot used to
- # create the snapshot, if cloud-init has not run
- boot_clean_script: |
- #!/bin/bash
- rm -rf /var/log/cloud-init.log /var/log/cloud-init-output.log \
- /var/lib/cloud/ /run/cloud-init/ /var/log/syslog
+ # global default configuration options
+ default:
+ # all are disabled by default
+ enabled: false
+ # timeout for booting image and running cloud init
+ boot_timeout: 120
+ # a script to run after a boot that is used to modify an image, before
+ # making a snapshot of the image. may be useful for removing data left
+ # behind from cloud-init booting, such as logs, to ensure that data
+ # from snapshot.launch() will not include a cloud-init.log from a boot
+ # used to create the snapshot, if cloud-init has not run
+ boot_clean_script: |
+ #!/bin/bash
+ rm -rf /var/log/cloud-init.log /var/log/cloud-init-output.log \
+ /var/lib/cloud/ /run/cloud-init/ /var/log/syslog
+ # test script to determine if system is booted fully
+ system_ready_script: |
+ # permit running or degraded state as both indicate complete boot
+ [ $(systemctl is-system-running) = 'running' -o
+ $(systemctl is-system-running) = 'degraded' ]
+ # test script to determine if cloud-init has finished
+ cloud_init_ready_script: |
+ [ -f '/run/cloud-init/result.json' ]
+
+ # lxd specific default configuration options
+ lxd:
+ # default sstreams server to use for lxd image retrieval
+ sstreams_server: https://us.images.linuxcontainers.org:8443
+ # keep base image, avoids downloading again next run
+ cache_base_image: true
+ # lxd images from linuxcontainers.org do not have the nocloud seed
+ # templates in place, so the image metadata must be modified
+ override_templates: true
+ # arg overrides to set image up
+ setup_overrides:
+ # lxd images from linuxcontainers.org do not come with
+ # cloud-init, so must pull cloud-init in from repo using
+ # setup_image.upgrade
+ upgrade: true
releases:
- trusty:
- enabled: true
- platform_ident:
- lxd:
- # if sstreams_server is omitted, default is used, defined in
- # tests.cloud_tests.platforms.lxd.DEFAULT_SSTREAMS_SERVER as:
- # sstreams_server: https://us.images.linuxcontainers.org:8443
- #alias: ubuntu/trusty/default
- alias: t
- sstreams_server: https://cloud-images.ubuntu.com/daily
- xenial:
- enabled: true
- platform_ident:
- lxd:
- #alias: ubuntu/xenial/default
- alias: x
- sstreams_server: https://cloud-images.ubuntu.com/daily
- yakkety:
- enabled: true
- platform_ident:
- lxd:
- #alias: ubuntu/yakkety/default
- alias: y
- sstreams_server: https://cloud-images.ubuntu.com/daily
+ # UBUNTU =================================================================
zesty:
- enabled: true
- platform_ident:
- lxd:
- #alias: ubuntu/zesty/default
- alias: z
- sstreams_server: https://cloud-images.ubuntu.com/daily
- jessie:
- platform_ident:
- lxd:
- alias: debian/jessie/default
+ # EOL: Jan 2018
+ default:
+ enabled: true
+ lxd:
+ sstreams_server: https://cloud-images.ubuntu.com/daily
+ alias: zesty
+ setup_overrides: null
+ override_templates: false
+ yakkety:
+ # EOL: Jul 2017
+ default:
+ enabled: true
+ lxd:
+ sstreams_server: https://cloud-images.ubuntu.com/daily
+ alias: yakkety
+ setup_overrides: null
+ override_templates: false
+ xenial:
+ # EOL: Apr 2021
+ default:
+ enabled: true
+ lxd:
+ sstreams_server: https://cloud-images.ubuntu.com/daily
+ alias: xenial
+ setup_overrides: null
+ override_templates: false
+ trusty:
+ # EOL: Apr 2019
+ default:
+ enabled: true
+ system_ready_script: |
+ #!/bin/bash
+ # upstart based, so use old style runlevels
+ [ $(runlevel | awk '{print $2}') = '2' ]
+ lxd:
+ sstreams_server: https://cloud-images.ubuntu.com/daily
+ alias: trusty
+ setup_overrides: null
+ override_templates: false
+ precise:
+ # EOL: Apr 2017
+ default:
+ # still supported but not relevant for development, not enabled
+ # tests should still work though unless they use newer features
+ enabled: false
+ system_ready_script: |
+ #!/bin/bash
+ # upstart based, so use old style runlevels
+ [ $(runlevel | awk '{print $2}') = '2' ]
+ lxd:
+ sstreams_server: https://cloud-images.ubuntu.com/daily
+ alias: precise
+ setup_overrides: null
+ override_templates: false
+ # DEBIAN =================================================================
sid:
- platform_ident:
- lxd:
- alias: debian/sid/default
+ # EOL: N/A
+ default:
+ # tests should work on sid, however it is not always stable
+ enabled: false
+ lxd:
+ alias: debian/sid/default
stretch:
- platform_ident:
- lxd:
- alias: debian/stretch/default
+ # EOL: Not yet released
+ default:
+ enabled: true
+ lxd:
+ alias: debian/stretch/default
+ jessie:
+ # EOL: Jun 2020
+ default:
+ enabled: true
+ lxd:
+ alias: debian/jessie/default
wheezy:
- platform_ident:
- lxd:
- alias: debian/wheezy/default
+ # EOL: May 2018 (Apr 2016 - end of full updates)
+ default:
+ # this is old enough that it is no longer relevant for development
+ enabled: false
+ lxd:
+ alias: debian/wheezy/default
+ # CENTOS =================================================================
centos70:
- timeout: 180
- platform_ident:
- lxd:
- alias: centos/7/default
+ # EOL: Jun 2024 (2020 - end of full updates)
+ default:
+ enabled: true
+ lxd:
+ alias: centos/7/default
centos66:
- timeout: 180
- platform_ident:
- lxd:
- alias: centos/6/default
+ # EOL: Nov 2020
+ default:
+ enabled: true
+ # still supported, but only bugfixes after may 2017
+ system_ready_script: |
+ #!/bin/bash
+ [ $(runlevel | awk '{print $2}') = '3' ]
+ lxd:
+ alias: centos/6/default
# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/setup_image.py b/tests/cloud_tests/setup_image.py
index 5d6c638..28cbfbf 100644
--- a/tests/cloud_tests/setup_image.py
+++ b/tests/cloud_tests/setup_image.py
@@ -7,6 +7,30 @@ from functools import partial
import os
+def installed_version(image, package, ensure_installed=True):
+ """
+ get installed version of package
+ image: cloud_tests.images instance to operate on
+ package: name of package
+ ensure_installed: raise error if not installed
+ return_value: cloud-init version string
+ """
+ # get right cmd for os family
+ os_family = util.get_os_family(image.properties['os'])
+ if os_family == 'debian':
+ cmd = ['dpkg-query', '-W', "--showformat='${Version}'", package]
+ elif os_family == 'redhat':
+ cmd = ['rpm', '-q', '--queryformat', "'%{VERSION}'", package]
+ else:
+ raise NotImplementedError
+
+ # query version
+ msg = 'query version for package: {}'.format(package)
+ (out, err, exit) = image.execute(cmd, description=msg,
+ ignore_errors=not ensure_installed)
+ return out.strip()
+
+
def install_deb(args, image):
"""
install deb into image
@@ -21,20 +45,18 @@ def install_deb(args, image):
'family: {}'.format(args.deb, os_family))
# install deb
- LOG.debug('installing deb: %s into target', args.deb)
+ msg = 'install deb: "{}" into target'.format(args.deb)
+ LOG.debug(msg)
remote_path = os.path.join('/tmp', os.path.basename(args.deb))
image.push_file(args.deb, remote_path)
- (out, err, exit) = image.execute(['dpkg', '-i', remote_path])
- if exit != 0:
- raise OSError('failed install deb: {}\n\tstdout: {}\n\tstderr: {}'
- .format(args.deb, out, err))
+ cmd = 'dpkg -i {} || apt-get install --yes -f'.format(remote_path)
+ image.execute(['/bin/sh', '-c', cmd], description=msg)
# check installed deb version matches package
fmt = ['-W', "--showformat='${Version}'"]
(out, err, exit) = image.execute(['dpkg-deb'] + fmt + [remote_path])
expected_version = out.strip()
- (out, err, exit) = image.execute(['dpkg-query'] + fmt + ['cloud-init'])
- found_version = out.strip()
+ found_version = installed_version(image, 'cloud-init')
if expected_version != found_version:
raise OSError('install deb version "{}" does not match expected "{}"'
.format(found_version, expected_version))
@@ -52,24 +74,21 @@ def install_rpm(args, image):
"""
# ensure system is compatible with package format
os_family = util.get_os_family(image.properties['os'])
- if os_family not in ['redhat', 'sles']:
+ if os_family != 'redhat':
raise NotImplementedError('install rpm: {} not supported on os '
'family: {}'.format(args.rpm, os_family))
# install rpm
- LOG.debug('installing rpm: %s into target', args.rpm)
+ msg = 'install rpm: "{}" into target'.format(args.rpm)
+ LOG.debug(msg)
remote_path = os.path.join('/tmp', os.path.basename(args.rpm))
image.push_file(args.rpm, remote_path)
- (out, err, exit) = image.execute(['rpm', '-U', remote_path])
- if exit != 0:
- raise OSError('failed to install rpm: {}\n\tstdout: {}\n\tstderr: {}'
- .format(args.rpm, out, err))
+ image.execute(['rpm', '-U', remote_path], description=msg)
fmt = ['--queryformat', '"%{VERSION}"']
(out, err, exit) = image.execute(['rpm', '-q'] + fmt + [remote_path])
expected_version = out.strip()
- (out, err, exit) = image.execute(['rpm', '-q'] + fmt + ['cloud-init'])
- found_version = out.strip()
+ found_version = installed_version(image, 'cloud-init')
if expected_version != found_version:
raise OSError('install rpm version "{}" does not match expected "{}"'
.format(found_version, expected_version))
@@ -80,13 +99,34 @@ def install_rpm(args, image):
def upgrade(args, image):
"""
- run the system's upgrade command
+ upgrade or install cloud-init from repo
+ args: cmdline arguments
+ image: cloud_tests.images instance to operate on
+ return_value: None, may raise errors
+ """
+ # determine command for os_family
+ os_family = util.get_os_family(image.properties['os'])
+ if os_family == 'debian':
+ cmd = 'apt-get update && apt-get install cloud-init --yes'
+ elif os_family == 'redhat':
+ cmd = 'yum install cloud-init --assumeyes'
+ else:
+ raise NotImplementedError
+
+ # upgrade cloud-init
+ msg = 'upgrading cloud-init'
+ LOG.debug(msg)
+ image.execute(['/bin/sh', '-c', cmd], description=msg)
+
+
+def upgrade_full(args, image):
+ """
+ run the system's full upgrade command
args: cmdline arguments
image: cloud_tests.images instance to operate on
return_value: None, may raise errors
"""
# determine appropriate upgrade command for os_family
- # TODO: maybe use cloudinit.distros for this?
os_family = util.get_os_family(image.properties['os'])
if os_family == 'debian':
cmd = 'apt-get update && apt-get upgrade --yes'
@@ -97,11 +137,9 @@ def upgrade(args, image):
'from family: {}'.format(os_family))
# upgrade system
- LOG.debug('upgrading system')
- (out, err, exit) = image.execute(['/bin/sh', '-c', cmd])
- if exit != 0:
- raise OSError('failed to upgrade system\n\tstdout: {}\n\tstderr:{}'
- .format(out, err))
+ msg = 'full system upgrade'
+ LOG.debug(msg)
+ image.execute(['/bin/sh', '-c', cmd], description=msg)
def run_script(args, image):
@@ -111,9 +149,9 @@ def run_script(args, image):
image: cloud_tests.images instance to operate on
return_value: None, may raise errors
"""
- # TODO: get exit status back from script and add error handling here
- LOG.debug('running setup image script in target image')
- image.run_script(args.script)
+ msg = 'run setup image script in target image'
+ LOG.debug(msg)
+ image.run_script(args.script, description=msg)
def enable_ppa(args, image):
@@ -124,17 +162,15 @@ def enable_ppa(args, image):
return_value: None, may raise errors
"""
# ppa only supported on ubuntu (maybe debian?)
- if image.properties['os'] != 'ubuntu':
+ if image.properties['os'].lower() != 'ubuntu':
raise NotImplementedError('enabling a ppa is only available on ubuntu')
# add ppa with add-apt-repository and update
ppa = 'ppa:{}'.format(args.ppa)
- LOG.debug('enabling %s', ppa)
+ msg = 'enable ppa: "{}" in target'.format(ppa)
+ LOG.debug(msg)
cmd = 'add-apt-repository --yes {} && apt-get update'.format(ppa)
- (out, err, exit) = image.execute(['/bin/sh', '-c', cmd])
- if exit != 0:
- raise OSError('enable ppa for {} failed\n\tstdout: {}\n\tstderr: {}'
- .format(ppa, out, err))
+ image.execute(['/bin/sh', '-c', cmd], description=msg)
def enable_repo(args, image):
@@ -155,11 +191,9 @@ def enable_repo(args, image):
raise NotImplementedError('enable repo command not configured for '
'distro from family: {}'.format(os_family))
- LOG.debug('enabling repo: "%s"', args.repo)
- (out, err, exit) = image.execute(['/bin/sh', '-c', cmd])
- if exit != 0:
- raise OSError('enable repo {} failed\n\tstdout: {}\n\tstderr: {}'
- .format(args.repo, out, err))
+ msg = 'enable repo: "{}" in target'.format(args.repo)
+ LOG.debug(msg)
+ image.execute(['/bin/sh', '-c', cmd], description=msg)
def setup_image(args, image):
@@ -169,6 +203,11 @@ def setup_image(args, image):
image: cloud_tests.image instance to operate on
return_value: tuple of results and fail count
"""
+ # update the args if necessary for this image
+ overrides = image.setup_overrides
+ LOG.debug('updating args for setup with: %s', overrides)
+ args = util.update_args(args, overrides, preserve_old=True)
+
# mapping of setup cmdline arg name to setup function
# represented as a tuple rather than a dict or odict as lookup by name not
# needed, and order is important as --script and --upgrade go at the end
@@ -179,17 +218,19 @@ def setup_image(args, image):
('repo', enable_repo, 'setup func for --repo, enable repo'),
('ppa', enable_ppa, 'setup func for --ppa, enable ppa'),
('script', run_script, 'setup func for --script, run script'),
- ('upgrade', upgrade, 'setup func for --upgrade, upgrade pkgs'),
+ ('upgrade', upgrade, 'setup func for --upgrade, upgrade cloud-init'),
+ ('upgrade-full', upgrade_full, 'setup func for --upgrade-full'),
)
# determine which setup functions needed
calls = [partial(stage.run_single, desc, partial(func, args, image))
for name, func, desc in handlers if getattr(args, name, None)]
- image_name = 'image: distro={}, release={}'.format(
- image.properties['os'], image.properties['release'])
- LOG.info('setting up %s', image_name)
- return stage.run_stage('set up for {}'.format(image_name), calls,
- continue_after_error=False)
+ LOG.info('setting up %s', image)
+ res = stage.run_stage(
+ 'set up for {}'.format(image), calls, continue_after_error=False)
+ LOG.debug('after setup complete, installed cloud-init version is: %s',
+ installed_version(image, 'cloud-init'))
+ return res
# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/snapshots/base.py b/tests/cloud_tests/snapshots/base.py
index d715f03..e9b721e 100644
--- a/tests/cloud_tests/snapshots/base.py
+++ b/tests/cloud_tests/snapshots/base.py
@@ -7,10 +7,11 @@ class Snapshot(object):
"""
platform_name = None
- def __init__(self, properties, config):
+ def __init__(self, platform, properties, config):
"""
Set up snapshot
"""
+ self.platform = platform
self.properties = properties
self.config = config
diff --git a/tests/cloud_tests/snapshots/lxd.py b/tests/cloud_tests/snapshots/lxd.py
index eabbce3..d2a123a 100644
--- a/tests/cloud_tests/snapshots/lxd.py
+++ b/tests/cloud_tests/snapshots/lxd.py
@@ -9,13 +9,12 @@ class LXDSnapshot(base.Snapshot):
"""
platform_name = "lxd"
- def __init__(self, properties, config, platform, pylxd_frozen_instance):
+ def __init__(self, platform, properties, config, pylxd_frozen_instance):
"""
Set up snapshot
"""
- self.platform = platform
self.pylxd_frozen_instance = pylxd_frozen_instance
- super(LXDSnapshot, self).__init__(properties, config)
+ super(LXDSnapshot, self).__init__(platform, properties, config)
def launch(self, user_data, meta_data=None, block=True, start=True,
use_desc=None):
@@ -34,10 +33,11 @@ class LXDSnapshot(base.Snapshot):
if meta_data:
inst_config['user.meta-data'] = meta_data
instance = self.platform.launch_container(
- container=self.pylxd_frozen_instance.name, config=inst_config,
- block=block, image_desc=str(self), use_desc=use_desc)
+ self.properties, self.config, block=block, image_desc=str(self),
+ container=self.pylxd_frozen_instance.name, use_desc=use_desc,
+ container_config=inst_config)
if start:
- instance.start(wait=True, wait_time=self.config.get('timeout'))
+ instance.start()
return instance
def destroy(self):
diff --git a/tests/cloud_tests/util.py b/tests/cloud_tests/util.py
index 64a8667..295fb25 100644
--- a/tests/cloud_tests/util.py
+++ b/tests/cloud_tests/util.py
@@ -1,5 +1,6 @@
# This file is part of cloud-init. See LICENSE file for license information.
+import copy
import glob
import os
import random
@@ -7,10 +8,18 @@ import string
import tempfile
import yaml
-from cloudinit.distros import OSFAMILIES
from cloudinit import util as c_util
from tests.cloud_tests import LOG
+OS_FAMILY_MAPPING = {
+ 'debian': ['debian', 'ubuntu'],
+ 'redhat': ['centos', 'rhel', 'fedora'],
+ 'gentoo': ['gentoo'],
+ 'freebsd': ['freebsd'],
+ 'suse': ['sles'],
+ 'arch': ['arch'],
+}
+
def list_test_data(data_dir):
"""
@@ -68,7 +77,7 @@ def gen_instance_name(prefix='cloud-test', image_desc=None, use_desc=None,
"""
filter bad characters out of elem and trim to length
"""
- elem = elem[:max_len] if elem else unknown
+ elem = elem.lower()[:max_len] if elem else unknown
return ''.join(c if c in valid else delim for c in elem)
return next(name for name in
@@ -88,7 +97,9 @@ def get_os_family(os_name):
"""
get os family type for os_name
"""
- return next((k for k, v in OSFAMILIES.items() if os_name in v), None)
+ os_name = os_name.lower()
+ return next((k for k, v in OS_FAMILY_MAPPING.items()
+ if os_name.lower() in v), None)
def current_verbosity():
@@ -158,6 +169,81 @@ def write_file(*args, **kwargs):
"""
write a file using cloudinit.util.write_file
"""
- c_util.write_file(*args, **kwargs)
+ return c_util.write_file(*args, **kwargs)
+
+
+def read_conf(*args, **kwargs):
+ """
+ read configuration using cloudinit.util.read_conf
+ """
+ return c_util.read_conf(*args, **kwargs)
+
+
+def subp(*args, **kwargs):
+ """
+ execute a command on the system shell using cloudinit.util.subp
+ """
+ return c_util.subp(*args, **kwargs)
+
+
+def tmpdir(prefix='cloud_test_util_'):
+ return tempfile.mkdtemp(prefix=prefix)
+
+
+def rel_files(basedir):
+ """
+ list of files under directory by relative path, not including directories
+ return_value: list or relative paths
+ """
+ basedir = os.path.normpath(basedir)
+ return [path[len(basedir) + 1:] for path in
+ glob.glob(os.path.join(basedir, '**'), recursive=True)
+ if not os.path.isdir(path)]
+
+
+def flat_tar(output, basedir, owner='root', group='root'):
+ """
+ create a flat tar archive (no leading ./) from basedir
+ output: output tar file to write
+ basedir: base directory for archive
+ owner: owner of archive files
+ group: group archive files belong to
+ return_value: none
+ """
+ c_util.subp(['tar', 'cf', output, '--owner', owner, '--group', group,
+ '-C', basedir] + rel_files(basedir), capture=True)
+
+
+def update_args(args, updates, preserve_old=True):
+ """
+ update cmdline arguments from a dictionary
+ args: cmdline arguments
+ updates: dictionary of {arg_name: new_value} mappings
+ preserve_old: if true, create a deep copy of args before updating
+ return_value: updated cmdline arguments, as new object if preserve_old=True
+ """
+ args = copy.deepcopy(args) if preserve_old else args
+ if updates:
+ vars(args).update(updates)
+ return args
+
+
+class InTargetExecuteError(c_util.ProcessExecutionError):
+ """
+ Error type for in target commands that fail
+ """
+ default_desc = 'Unexpected error while running command in target instance'
+
+ def __init__(self, stdout, stderr, exit_code, cmd, instance,
+ description=None):
+ """
+ init error and parent error class
+ """
+ if isinstance(cmd, (tuple, list)):
+ cmd = ' '.join(cmd)
+ super(InTargetExecuteError, self).__init__(
+ stdout=stdout, stderr=stderr, exit_code=exit_code, cmd=cmd,
+ reason="Instance: {}".format(instance),
+ description=description if description else self.default_desc)
# vi: ts=4 expandtab
diff --git a/tox.ini b/tox.ini
index ca5d8b8..fe194de 100644
--- a/tox.ini
+++ b/tox.ini
@@ -64,6 +64,20 @@ deps =
flake8==2.5.4
hacking==0.10.2
+[testenv:citest]
+basepython = python3
+commands = {envpython} -m tests.cloud_tests {posargs}
+passenv = HOME
+deps =
+ pylxd==2.1.3
+
+[testenv:citest_new_pylxd]
+basepython = python3
+commands = {envpython} -m tests.cloud_tests {posargs}
+passenv = HOME
+deps =
+ pylxd==2.2.2
+
[testenv:centos6]
basepython = python2.6
commands = nosetests {posargs:tests}
Follow ups
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Scott Moser, 2017-06-14
-
[Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Scott Moser, 2017-06-14
-
[Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Scott Moser, 2017-05-24
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Scott Moser, 2017-05-24
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-17
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-15
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-15
-
[Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Wesley Wiedenmeier, 2017-03-15
-
[Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Wesley Wiedenmeier, 2017-03-15
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Wesley Wiedenmeier, 2017-03-15
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-13
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-13
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-13
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-12
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-12
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-12
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-08
-
Re: [Merge] ~wesley-wiedenmeier/cloud-init:integration-testing into cloud-init:master
From: Server Team CI bot, 2017-03-06