cloud-init-dev team mailing list archive
-
cloud-init-dev team
-
Mailing list archive
-
Message #03961
[Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
Joshua Powers has proposed merging ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master.
Commit message:
tests: Enable AWS EC2 Integration Testing
This enables integration tests to utilize AWS EC2 as a testing platform.
This is done through the usage of the boto3 library.
Usage will create a custom VPC if one does not exist with the ec2 tag
name of 'cloud-init-testing' by default. The VPC is setup with both IPv4
and IPv6 capabilities, but will only hand out IPv4 addresses by default.
Instances will have complete internet access, but only allow SSH
ingress.
SSH keys are generated with each run of the integration tests with the
key getting uploaded to AWS at the start of tests and deleted on exit.
To enable creation when the platform is setup the SSH generation code is
moved to be completed by the platform setup and not during image setup.
The nocloud-kvm platform was updated with this change.
Creating a custom image will utilize the same clean script,
boot_clean_script, that the LXD platform uses as well. The custom AMI
is generated, used, and de-registered after a test run.
The default instance type is set to t2.micro.
The default timeout for ec2 was increased to 300 from 120 as many tests
hit up against the 2 minute timeout and depending on region load can
go over.
Documentation for the AWS platform was added with the expected
configuration files for the platform to be used.
pylint exception was added for paramiko, which until now, was not
flagged and boto3 was added to the list of dependencies in the tox
runner.
Finally, the apt_configure_sources_list testcase was made more generic
as cloud vendors use custom sources.list that did not meet our regex.
Requested reviews:
cloud-init commiters (cloud-init-dev)
For more details, see:
https://code.launchpad.net/~powersj/cloud-init/+git/cloud-init/+merge/335186
All Tests on Bionic + Xenial with image version of cloud-init
---
ec2: https://paste.ubuntu.com/26173562/
ec2: https://paste.ubuntu.com/26179600/ (bionic only)
lxd: https://paste.ubuntu.com/26178523/
nocloud-kvm: busted daily image
$ python3 -m tests.cloud_tests run --os-name bionic --os-name xenial --platform ec2 --preserve-data --data-dir results_ec2 --verbose
$ python3 -m tests.cloud_tests run --os-name bionic --os-name xenial --platform lxd --preserve-data --data-dir results_lxd --verbose
$ python3 -m tests.cloud_tests run --os-name bionic --os-name xenial --platform nocloud-kvm --preserve-data --data-dir results_nocloud_kvm --verbose
All Tests on Bionic + Xenial with Bionic deb of cloud-init
---
ec2: https://paste.ubuntu.com/26179793/ (bionic only)
lxd: https://paste.ubuntu.com/26178405/
nocloud-kvm: https://paste.ubuntu.com/26178282/
$ python3 -m tests.cloud_tests run --os-name bionic --os-name xenial --platform ec2 --preserve-data --data-dir results_ec2 --verbose --deb cloud-init_17.1-60-ga30a3bb5-0ubuntu1_all.deb
$ python3 -m tests.cloud_tests run --os-name bionic --os-name xenial --platform lxd --preserve-data --data-dir results_lxd --verbose --deb cloud-init_17.1-60-ga30a3bb5-0ubuntu1_all.deb
$ python3 -m tests.cloud_tests run --os-name bionic --os-name xenial --platform nocloud-kvm --preserve-data --data-dir results_nocloud_kvm --verbose --deb cloud-init_17.1-60-ga30a3bb5-0ubuntu1_all.deb
--
Your team cloud-init commiters is requested to review the proposed merge of ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master.
diff --git a/.pylintrc b/.pylintrc
index 3ad3692..ce0388f 100644
--- a/.pylintrc
+++ b/.pylintrc
@@ -46,7 +46,7 @@ reports=no
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis. It
# supports qualified module names, as well as Unix pattern matching.
-ignored-modules=six.moves,pkg_resources,httplib,http.client
+ignored-modules=six.moves,pkg_resources,httplib,http.client,paramiko
# List of class names for which member attributes should not be checked (useful
# for classes with dynamically set attributes). This supports the use of
diff --git a/doc/rtd/topics/tests.rst b/doc/rtd/topics/tests.rst
index d668e3f..730df29 100644
--- a/doc/rtd/topics/tests.rst
+++ b/doc/rtd/topics/tests.rst
@@ -118,19 +118,19 @@ TreeRun and TreeCollect
If working on a cloud-init feature or resolving a bug, it may be useful to
run the current copy of cloud-init in the integration testing environment.
-The integration testing suite can automatically build a deb based on the
+The integration testing suite can automatically build a deb based on the
current working tree of cloud-init and run the test suite using this deb.
The ``tree_run`` and ``tree_collect`` commands take the same arguments as
-the ``run`` and ``collect`` commands. These commands will build a deb and
-write it into a temporary file, then start the test suite and pass that deb
+the ``run`` and ``collect`` commands. These commands will build a deb and
+write it into a temporary file, then start the test suite and pass that deb
in. To build a deb only, and not run the test suite, the ``bddeb`` command
can be used.
Note that code in the cloud-init working tree that has not been committed
when the cloud-init deb is built will still be included. To build a
cloud-init deb from or use the ``tree_run`` command using a copy of
-cloud-init located in a different directory, use the option ``--cloud-init
+cloud-init located in a different directory, use the option ``--cloud-init
/path/to/cloud-init``.
.. code-block:: bash
@@ -383,7 +383,7 @@ Development Checklist
* Valid unit tests validating output collected
* Passes pylint & pep8 checks
* Placed in the appropriate sub-folder in the test cases directory
-* Tested by running the test:
+* Tested by running the test:
.. code-block:: bash
@@ -392,6 +392,32 @@ Development Checklist
--test modules/your_test.yaml \
[--deb <build of cloud-init>]
+
+Platforms
+=========
+
+EC2
+---
+To run on the EC2 platform it is required that the user has an AWS credentials
+configuration file specifying his or her access keys and a default region.
+These configuration files are the standard that the AWS cli and other AWS
+tools utilize for interacting directly with AWS itself and are normally
+generated when running ``aws configure``:
+
+.. code-block:: bash
+
+ $ cat $HOME/.aws/config
+ [default]
+ aws_access_key_id = <KEY HERE>
+ aws_secret_access_key = <KEY HERE>
+
+.. code-block:: bash
+
+ $ cat $HOME/.aws/config
+ [default]
+ region = us-west-2
+
+
Architecture
============
@@ -455,7 +481,7 @@ replace the default. If the data is a dictionary then the value will be the
result of merging that dictionary from the default config and that
dictionary from the overrides.
-Merging is done using the function
+Merging is done using the function
``tests.cloud_tests.config.merge_config``, which can be examined for more
detail on config merging behavior.
diff --git a/tests/cloud_tests/platforms.yaml b/tests/cloud_tests/platforms.yaml
index fa4f845..458ab81 100644
--- a/tests/cloud_tests/platforms.yaml
+++ b/tests/cloud_tests/platforms.yaml
@@ -6,8 +6,13 @@ default_platform_config:
get_image_timeout: 300
# maximum time to create instance (before waiting for cloud-init)
create_instance_timeout: 60
-
+ private_key: id_rsa
+ public_key: id_rsa.pub
platforms:
+ ec2:
+ enabled: true
+ instance-type: t2.micro
+ tag: cloud-init-testing
lxd:
enabled: true
# overrides for image templates
@@ -61,9 +66,5 @@ platforms:
{{ config_get("user.vendor-data", properties.default) }}
nocloud-kvm:
enabled: true
- private_key: id_rsa
- public_key: id_rsa.pub
- ec2: {}
- azure: {}
# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/platforms/__init__.py b/tests/cloud_tests/platforms/__init__.py
index 92ed162..a01e51a 100644
--- a/tests/cloud_tests/platforms/__init__.py
+++ b/tests/cloud_tests/platforms/__init__.py
@@ -2,10 +2,12 @@
"""Main init."""
+from .ec2 import platform as ec2
from .lxd import platform as lxd
from .nocloudkvm import platform as nocloudkvm
PLATFORMS = {
+ 'ec2': ec2.EC2Platform,
'nocloud-kvm': nocloudkvm.NoCloudKVMPlatform,
'lxd': lxd.LXDPlatform,
}
diff --git a/tests/cloud_tests/platforms/ec2/image.py b/tests/cloud_tests/platforms/ec2/image.py
new file mode 100644
index 0000000..456dda2
--- /dev/null
+++ b/tests/cloud_tests/platforms/ec2/image.py
@@ -0,0 +1,105 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+"""EC2 Image Base Class."""
+
+from ..images import Image
+from .snapshot import EC2Snapshot
+
+
+class EC2Image(Image):
+ """EC2 backed image."""
+
+ platform_name = 'ec2'
+
+ def __init__(self, platform, config, image_ami):
+ """Set up image.
+
+ @param platform: platform object
+ @param config: image configuration
+ @param image_ami: string of image ami ID
+ """
+ super(EC2Image, self).__init__(platform, config)
+ self._img_instance = None
+ self.image_ami = image_ami
+ self.image_ami_edited = False
+
+ @property
+ def _instance(self):
+ """Internal use only, returns a running instance"""
+ if not self._img_instance:
+ self._img_instance = self.platform.create_instance(
+ self.properties, self.config, self.features,
+ self.image_ami, b'')
+ self._img_instance.start(wait=True, wait_for_cloud_init=True)
+ self.modified = True
+ return self._img_instance
+
+ @property
+ def properties(self):
+ """Dictionary containing: 'arch', 'os', 'version', 'release'."""
+ return {
+ 'arch': self.config['arch'],
+ 'os': self.config['family'],
+ 'release': self.config['release'],
+ 'version': self.config['version'],
+ }
+
+ def destroy(self):
+ """Unset path to signal image is no longer used.
+
+ The removal of the images and all other items is handled by the
+ framework. In some cases we want to keep the images, so let the
+ framework decide whether to keep or destroy everything.
+ """
+ if self.image_ami_edited:
+ self.platform.ec2_client.deregister_image(
+ ImageId=self.image_ami_edited
+ )
+
+ super(EC2Image, self).destroy()
+
+ def _execute(self, *args, **kwargs):
+ """Execute command in image, modifying image."""
+ return self._instance._execute(*args, **kwargs)
+
+ def push_file(self, local_path, remote_path):
+ """Copy file at 'local_path' to instance at 'remote_path'."""
+ return self._instance.push_file(local_path, remote_path)
+
+ def run_script(self, *args, **kwargs):
+ """Run script in image, modifying image.
+
+ @return_value: script output
+ """
+ return self._instance.run_script(*args, **kwargs)
+
+ def snapshot(self):
+ """Create snapshot of image, block until done.
+
+ Will return base image_ami if no instance has been booted, otherwise
+ will run the clean script, shutdown the instance, create a custom
+ AMI, and use that AMI once available.
+ """
+ if not self._img_instance:
+ return EC2Snapshot(self.platform, self.properties, self.config,
+ self.features, self.image_ami)
+
+ if self.config.get('boot_clean_script'):
+ self._img_instance.run_script(self.config.get('boot_clean_script'))
+
+ self._img_instance.shutdown(wait=True)
+
+ name = '%s-%s-%s' % (
+ self.platform.tag, self.image_ami, self.platform.uuid
+ )
+ response = self.platform.ec2_client.create_image(
+ Name=name, InstanceId=self._img_instance.instance.instance_id
+ )
+ self.image_ami_edited = response['ImageId']
+ image = self.platform.ec2_resource.Image(self.image_ami_edited)
+ self.platform.wait_for_state(image, 'available')
+
+ return EC2Snapshot(self.platform, self.properties, self.config,
+ self.features, self.image_ami_edited)
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/platforms/ec2/instance.py b/tests/cloud_tests/platforms/ec2/instance.py
new file mode 100644
index 0000000..72bafee
--- /dev/null
+++ b/tests/cloud_tests/platforms/ec2/instance.py
@@ -0,0 +1,105 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+"""Base EC2 instance."""
+import os
+
+import botocore
+
+from ..instances import Instance
+from tests.cloud_tests import util
+
+
+class EC2Instance(Instance):
+ """EC2 backed instance."""
+
+ platform_name = "ec2"
+ _ssh_client = None
+
+ def __init__(self, platform, properties, config, features,
+ image_ami, user_data):
+ """Set up instance.
+
+ @param platform: platform object
+ @param properties: dictionary of properties
+ @param config: dictionary of configuration values
+ @param features: dictionary of supported feature flags
+ @param image_ami: AWS AMI ID for image to use
+ @param user_data: test user-data to pass to instance
+ """
+ super(EC2Instance, self).__init__(
+ platform, image_ami, properties, config, features)
+
+ self.image_ami = image_ami
+ self.instance = None
+ self.user_data = user_data
+ self.ssh_ip = None
+ self.ssh_port = 22
+ self.ssh_key_file = os.path.join(
+ platform.config['data_dir'], platform.config['private_key'])
+ self.ssh_pubkey_file = os.path.join(
+ platform.config['data_dir'], platform.config['public_key'])
+
+ def destroy(self):
+ """Clean up instance."""
+ if self._ssh_client:
+ self._ssh_client.close()
+ self._ssh_client = None
+
+ if self.instance:
+ self.instance.terminate()
+
+ super(EC2Instance, self).destroy()
+
+ def _execute(self, command, stdin=None, env=None):
+ """Execute command on instance."""
+ env_args = []
+ if env:
+ env_args = ['env'] + ["%s=%s" for k, v in env.items()]
+
+ return self._ssh(['sudo'] + env_args + list(command), stdin=stdin)
+
+ def start(self, wait=True, wait_for_cloud_init=False):
+ """Start instance on EC2 with the platfrom's VPC."""
+ name_tag = {
+ 'ResourceType': 'instance',
+ 'Tags': [
+ {
+ 'Key': 'Name',
+ 'Value': self.platform.tag,
+ },
+ ]
+ }
+
+ try:
+ instances = self.platform.ec2_resource.create_instances(
+ ImageId=self.image_ami,
+ InstanceType=self.platform.instance_type,
+ KeyName=self.platform.key_name,
+ MaxCount=1,
+ MinCount=1,
+ SecurityGroupIds=[self.platform.vpc_security_group.id],
+ SubnetId=self.platform.vpc_subnet.id,
+ TagSpecifications=[name_tag],
+ UserData=self.user_data,
+ )
+ except botocore.exceptions.ClientError as error:
+ error_msg = error.response['Error']['Message']
+ raise util.InTargetExecuteError(b'', b'', 1, '', 'start',
+ reason=error_msg)
+
+ self.instance = instances[0]
+
+ if wait:
+ # Need to give AWS some time to process the launch
+ self.platform.wait_for_state(self.instance, 'running')
+ self.ssh_ip = self.instance.public_ip_address
+ self._wait_for_system(wait_for_cloud_init)
+
+ def shutdown(self, wait=True):
+ """Shutdown instance."""
+ self.instance.stop()
+
+ if wait:
+ self.platform.wait_for_state(self.instance, 'stopped')
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/platforms/ec2/platform.py b/tests/cloud_tests/platforms/ec2/platform.py
new file mode 100644
index 0000000..8ff2716
--- /dev/null
+++ b/tests/cloud_tests/platforms/ec2/platform.py
@@ -0,0 +1,255 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+"""Base EC2 platform."""
+import os
+import time
+import uuid
+
+import boto3
+import botocore
+
+from ..platforms import Platform
+from .image import EC2Image
+from .instance import EC2Instance
+from tests.cloud_tests import util
+
+
+class EC2Platform(Platform):
+ """EC2 test platform."""
+
+ platform_name = 'ec2'
+ ipv4_cidr = '192.168.1.0/20'
+
+ def __init__(self, config):
+ """Set up platform."""
+ super(EC2Platform, self).__init__(config)
+ # Used with SSH key and custom AMI generation naming with EC2
+ self.uuid = str(uuid.uuid1())[0:8]
+ self.tag = config['tag']
+
+ self.ec2_client = boto3.client('ec2')
+ self.ec2_resource = boto3.resource('ec2')
+ self.instance_type = config['instance-type']
+ self.key_name = self._upload_public_key(config)
+ self.vpc = self._setup_vpc()
+ self.vpc_security_group = self._get_security_group()
+ self.vpc_subnet = self._get_subnet()
+
+ def create_instance(self, properties, config, features,
+ image_ami, user_data):
+ """Create an instance
+
+ @param src_img_path: image path to launch from
+ @param properties: image properties
+ @param config: image configuration
+ @param features: image features
+ @param image_ami: string of image ami ID
+ @param user_data: test user-data to pass to instance
+ @return_value: cloud_tests.instances instance
+ """
+ return EC2Instance(self, properties, config, features,
+ image_ami, user_data)
+
+ def destroy(self):
+ """Delete platform."""
+ if self.key_name:
+ self.ec2_client.delete_key_pair(KeyName=self.key_name)
+
+ def get_image(self, img_conf):
+ """Get image using specified image configuration.
+
+ @param img_conf: configuration for image
+ @return_value: cloud_tests.images instance
+ """
+ if img_conf['root-store'] == 'ebs':
+ image_type = 'hvm-ssd'
+ elif img_conf['root-store'] == 'instance-store':
+ image_type = 'hvm-instance'
+ else:
+ raise RuntimeError('Unknown root-store type: %s' %
+ (img_conf['root-store']))
+
+ image_filter = {
+ 'Name': 'name',
+ 'Values': ['ubuntu/images-testing/%s/ubuntu-%s-daily-%s-server-*'
+ % (image_type, img_conf['release'], img_conf['arch'])]
+ }
+ response = self.ec2_client.describe_images(Filters=[image_filter])
+ images = sorted(response['Images'], key=lambda k: k['CreationDate'])
+
+ try:
+ image_ami = images[-1]['ImageId']
+ except IndexError:
+ raise RuntimeError('No images found for %s!' % img_conf['release'])
+
+ image = EC2Image(self, img_conf, image_ami)
+ return image
+
+ @staticmethod
+ def wait_for_state(resource, goal_state):
+ """Wait up to 5 minutes (5 * 60) for a specific state.
+
+ @param: resource to check with resource.state
+ @param: string goal state to be in
+ """
+ attempts = 0
+
+ while attempts < 60:
+ try:
+ resource.reload()
+ current_state = resource.state
+ # Sometimes get {'Code': 0, 'Name': 'pending'}
+ if isinstance(current_state, dict):
+ current_state = current_state['Name']
+
+ if current_state == goal_state:
+ return
+ # will occur if instance is not up yet
+ except botocore.exception.BotoSeverError:
+ pass
+
+ time.sleep(5)
+ attempts += 1
+
+ raise util.InTargetExecuteError(
+ b'', b'', 1, goal_state, 'wait_for_state',
+ reason='%s: took too long to get to %s' % (resource, goal_state)
+ )
+
+ def _create_internet_gateway(self, vpc):
+ """Create Internet Gateway and assign to VPC.
+
+ @param vpc: VPC to create internet gateway on
+ """
+ internet_gateway = self.ec2_resource.create_internet_gateway()
+ internet_gateway.attach_to_vpc(VpcId=vpc.vpc_id)
+ self._tag_resource(internet_gateway)
+ return internet_gateway
+
+ def _create_subnet(self, vpc):
+ """Generate IPv4 and IPv6 subnets for use.
+
+ @param vpc: VPC to create subnets on
+ """
+ ipv6_block = vpc.ipv6_cidr_block_association_set[0]['Ipv6CidrBlock']
+ ipv6_cidr = ipv6_block[:-2] + '64'
+
+ subnet = vpc.create_subnet(CidrBlock=self.ipv4_cidr,
+ Ipv6CidrBlock=ipv6_cidr)
+ modify_subnet = subnet.meta.client.modify_subnet_attribute
+ modify_subnet(SubnetId=subnet.id,
+ MapPublicIpOnLaunch={'Value': True})
+ self._tag_resource(subnet)
+
+ def _get_security_group(self):
+ """Return default security group from VPC.
+
+ For now there should only be one.
+
+ @return_value: Return first security group object
+ """
+ return list(self.vpc.security_groups.all())[0]
+
+ def _get_subnet(self):
+ """Return first subnet from VPC.
+
+ For now there should only be one.
+
+ @return_value: Return subnet object
+ """
+ return list(self.vpc.subnets.all())[0]
+
+ def _setup_vpc(self):
+ """Setup AWS EC2 VPC or return existing VPC."""
+ for vpc in self.ec2_resource.vpcs.all():
+ if not vpc.tags:
+ continue
+ for tag in vpc.tags:
+ if tag['Value'] == self.tag:
+ return vpc
+
+ vpc = self.ec2_resource.create_vpc(
+ CidrBlock=self.ipv4_cidr,
+ AmazonProvidedIpv6CidrBlock=True
+ )
+ vpc.wait_until_available()
+ self._tag_resource(vpc)
+
+ internet_gateway = self._create_internet_gateway(vpc)
+ self._create_subnet(vpc)
+ self._update_routing_table(vpc, internet_gateway.id)
+ self._update_security_group(vpc)
+
+ return vpc
+
+ def _tag_resource(self, resource):
+ """Tag a resouce with the specified tag.
+
+ This makes finding and deleting resources specific to this testing
+ much easier to find.
+
+ @param resource: resource to tag"""
+ tag = {
+ 'Key': 'Name',
+ 'Value': self.tag
+ }
+ resource.create_tags(Tags=[tag])
+
+ def _update_routing_table(self, vpc, internet_gateway_id):
+ """Update default routing table with internet gateway.
+
+ This sets up internet access between the VPC via the internet gateway
+ by configuring routing tables for IPv4 and IPv6.
+
+ @param vpc: VPC containing routing table to update
+ @param internet_gateway_id: gateway ID to use for routing
+ """
+ route_table = list(vpc.route_tables.all())[0]
+ route_table.create_route(DestinationCidrBlock='0.0.0.0/0',
+ GatewayId=internet_gateway_id)
+ route_table.create_route(DestinationIpv6CidrBlock='::/0',
+ GatewayId=internet_gateway_id)
+ self._tag_resource(route_table)
+
+ def _update_security_group(self, vpc):
+ """Allow only SSH inbound to default VPC security group.
+
+ This revokes the initial accept all permissions and only allows
+ the SSH inbound.
+
+ @param vpc: VPC containing security group to update
+ """
+ ssh = {
+ 'IpProtocol': 'TCP',
+ 'FromPort': 22,
+ 'ToPort': 22,
+ 'IpRanges': [{'CidrIp': '0.0.0.0/0'}],
+ 'Ipv6Ranges': [{'CidrIpv6': '::/0'}]
+ }
+
+ security_group = list(vpc.security_groups.all())[0]
+ security_group.revoke_ingress(
+ IpPermissions=security_group.ip_permissions
+ )
+ security_group.authorize_ingress(
+ IpPermissions=[ssh]
+ )
+ self._tag_resource(security_group)
+
+ def _upload_public_key(self, config):
+ """Generate random name and upload SSH key with that name.
+
+ @param config: platform config
+ @return: string of ssh key name
+ """
+ key_file = os.path.join(config['data_dir'], config['public_key'])
+ with open(key_file, 'r') as file:
+ public_key = file.read().strip('\n')
+
+ name = '%s-%s' % (self.tag, self.uuid)
+ self.ec2_client.import_key_pair(KeyName=name,
+ PublicKeyMaterial=public_key)
+
+ return name
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/platforms/ec2/snapshot.py b/tests/cloud_tests/platforms/ec2/snapshot.py
new file mode 100644
index 0000000..a51c951
--- /dev/null
+++ b/tests/cloud_tests/platforms/ec2/snapshot.py
@@ -0,0 +1,51 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+
+"""Base EC2 snapshot."""
+
+from ..snapshots import Snapshot
+
+
+class EC2Snapshot(Snapshot):
+ """EC2 image copy backed snapshot."""
+
+ platform_name = 'ec2'
+
+ def __init__(self, platform, properties, config, features, image_ami):
+ """Set up snapshot.
+
+ @param platform: platform object
+ @param properties: image properties
+ @param config: image config
+ @param features: supported feature flags
+ @param image_ami: string of image ami ID
+ """
+ super(EC2Snapshot, self).__init__(
+ platform, properties, config, features)
+
+ self.image_ami = image_ami
+
+ def destroy(self):
+ """Clean up snapshot data."""
+ super(EC2Snapshot, self).destroy()
+
+ def launch(self, user_data, meta_data=None, block=True, start=True,
+ use_desc=None):
+ """Launch instance.
+
+ @param user_data: user-data for the instance
+ @param instance_id: instance-id for the instance
+ @param block: wait until instance is created
+ @param start: start instance and wait until fully started
+ @param use_desc: string of test name
+ @return_value: an Instance
+ """
+ instance = self.platform.create_instance(
+ self.properties, self.config, self.features,
+ self.image_ami, user_data)
+
+ if start:
+ instance.start()
+
+ return instance
+
+# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/platforms/instances.py b/tests/cloud_tests/platforms/instances.py
index 8c59d62..6918003 100644
--- a/tests/cloud_tests/platforms/instances.py
+++ b/tests/cloud_tests/platforms/instances.py
@@ -1,14 +1,22 @@
# This file is part of cloud-init. See LICENSE file for license information.
"""Base instance."""
+import time
+
+import paramiko
+from paramiko.ssh_exception import (BadHostKeyException,
+ AuthenticationException,
+ SSHException)
from ..util import TargetBase
+from tests.cloud_tests import util
class Instance(TargetBase):
"""Base instance object."""
platform_name = None
+ _ssh_client = None
def __init__(self, platform, name, properties, config, features):
"""Set up instance.
@@ -26,6 +34,10 @@ class Instance(TargetBase):
self.features = features
self._tmp_count = 0
+ self.ssh_ip = None
+ self.ssh_port = None
+ self.ssh_key_file = None
+
def console_log(self):
"""Instance console.
@@ -49,6 +61,58 @@ class Instance(TargetBase):
"""Clean up instance."""
pass
+ def _ssh(self, command, stdin=None):
+ """Run a command via SSH."""
+ client = self._ssh_connect()
+
+ cmd = util.shell_pack(command)
+ try:
+ fp_in, fp_out, fp_err = client.exec_command(cmd)
+ channel = fp_in.channel
+
+ if stdin is not None:
+ fp_in.write(stdin)
+ fp_in.close()
+
+ channel.shutdown_write()
+ rc = channel.recv_exit_status()
+ except SSHException as e:
+ raise util.InTargetExecuteError(b'', b'', 1, command, 'ssh',
+ reason=e)
+
+ return (fp_out.read(), fp_err.read(), rc)
+
+ def _ssh_connect(self):
+ """Connect via SSH."""
+ if self._ssh_client:
+ return self._ssh_client
+
+ if not self.ssh_ip or not self.ssh_port:
+ raise ValueError
+
+ client = paramiko.SSHClient()
+ client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
+ private_key = paramiko.RSAKey.from_private_key_file(self.ssh_key_file)
+
+ retries = 30
+ while retries:
+ try:
+ client.connect(username='ubuntu', hostname=self.ssh_ip,
+ port=self.ssh_port, pkey=private_key,
+ banner_timeout=30)
+ self._ssh_client = client
+ return client
+ except (ConnectionRefusedError, AuthenticationException,
+ BadHostKeyException, ConnectionResetError, SSHException,
+ OSError) as e:
+ retries -= 1
+ time.sleep(10)
+
+ ssh_cmd = 'Failed command to: ubuntu@%s:%s after 300 seconds' % (
+ self.ssh_ip, self.ssh_port
+ )
+ raise util.InTargetExecuteError(b'', b'', 1, ssh_cmd, 'ssh')
+
def _wait_for_system(self, wait_for_cloud_init):
"""Wait until system has fully booted and cloud-init has finished.
diff --git a/tests/cloud_tests/platforms/nocloudkvm/instance.py b/tests/cloud_tests/platforms/nocloudkvm/instance.py
index 9bb2425..e6f2dec 100644
--- a/tests/cloud_tests/platforms/nocloudkvm/instance.py
+++ b/tests/cloud_tests/platforms/nocloudkvm/instance.py
@@ -4,7 +4,6 @@
import copy
import os
-import paramiko
import socket
import subprocess
import time
@@ -26,7 +25,6 @@ class NoCloudKVMInstance(Instance):
"""NoCloud KVM backed instance."""
platform_name = "nocloud-kvm"
- _ssh_client = None
def __init__(self, platform, name, image_path, properties, config,
features, user_data, meta_data):
@@ -39,6 +37,10 @@ class NoCloudKVMInstance(Instance):
@param config: dictionary of configuration values
@param features: dictionary of supported feature flags
"""
+ super(NoCloudKVMInstance, self).__init__(
+ platform, name, properties, config, features
+ )
+
self.user_data = user_data
if meta_data:
meta_data = copy.deepcopy(meta_data)
@@ -66,6 +68,7 @@ class NoCloudKVMInstance(Instance):
meta_data['public-keys'] = []
meta_data['public-keys'].append(self.ssh_pubkey)
+ self.ssh_ip = '127.0.0.1'
self.ssh_port = None
self.pid = None
self.pid_file = None
@@ -73,9 +76,6 @@ class NoCloudKVMInstance(Instance):
self.disk = image_path
self.meta_data = meta_data
- super(NoCloudKVMInstance, self).__init__(
- platform, name, properties, config, features)
-
def destroy(self):
"""Clean up instance."""
if self.pid:
@@ -99,7 +99,7 @@ class NoCloudKVMInstance(Instance):
if env:
env_args = ['env'] + ["%s=%s" for k, v in env.items()]
- return self.ssh(['sudo'] + env_args + list(command), stdin=stdin)
+ return self._ssh(['sudo'] + env_args + list(command), stdin=stdin)
def generate_seed(self, tmpdir):
"""Generate nocloud seed from user-data"""
@@ -125,50 +125,6 @@ class NoCloudKVMInstance(Instance):
s.close()
return num
- def ssh(self, command, stdin=None):
- """Run a command via SSH."""
- client = self._ssh_connect()
-
- cmd = util.shell_pack(command)
- try:
- fp_in, fp_out, fp_err = client.exec_command(cmd)
- channel = fp_in.channel
- if stdin is not None:
- fp_in.write(stdin)
- fp_in.close()
-
- channel.shutdown_write()
- rc = channel.recv_exit_status()
- return (fp_out.read(), fp_err.read(), rc)
- except paramiko.SSHException as e:
- raise util.InTargetExecuteError(
- b'', b'', -1, command, self.name, reason=e)
-
- def _ssh_connect(self, hostname='localhost', username='ubuntu',
- banner_timeout=120, retry_attempts=30):
- """Connect via SSH."""
- if self._ssh_client:
- return self._ssh_client
-
- private_key = paramiko.RSAKey.from_private_key_file(self.ssh_key_file)
- client = paramiko.SSHClient()
- client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
- while retry_attempts:
- try:
- client.connect(hostname=hostname, username=username,
- port=self.ssh_port, pkey=private_key,
- banner_timeout=banner_timeout)
- self._ssh_client = client
- return client
- except (paramiko.SSHException, TypeError):
- time.sleep(1)
- retry_attempts = retry_attempts - 1
-
- error_desc = 'Failed command to: %s@%s:%s' % (username, hostname,
- self.ssh_port)
- raise util.InTargetExecuteError('', '', -1, 'ssh connect',
- self.name, error_desc)
-
def start(self, wait=True, wait_for_cloud_init=False):
"""Start instance."""
tmpdir = self.platform.config['data_dir']
diff --git a/tests/cloud_tests/platforms/nocloudkvm/platform.py b/tests/cloud_tests/platforms/nocloudkvm/platform.py
index 8593346..a7e6f5d 100644
--- a/tests/cloud_tests/platforms/nocloudkvm/platform.py
+++ b/tests/cloud_tests/platforms/nocloudkvm/platform.py
@@ -21,6 +21,10 @@ class NoCloudKVMPlatform(Platform):
platform_name = 'nocloud-kvm'
+ def __init__(self, config):
+ """Set up platform."""
+ super(NoCloudKVMPlatform, self).__init__(config)
+
def get_image(self, img_conf):
"""Get image using specified image configuration.
diff --git a/tests/cloud_tests/platforms/platforms.py b/tests/cloud_tests/platforms/platforms.py
index 2897536..93d6b88 100644
--- a/tests/cloud_tests/platforms/platforms.py
+++ b/tests/cloud_tests/platforms/platforms.py
@@ -1,6 +1,9 @@
# This file is part of cloud-init. See LICENSE file for license information.
"""Base platform class."""
+import os
+
+from cloudinit import util as c_util
class Platform(object):
@@ -11,6 +14,7 @@ class Platform(object):
def __init__(self, config):
"""Set up platform."""
self.config = config
+ self._generate_ssh_keys(config['data_dir'])
def get_image(self, img_conf):
"""Get image using specified image configuration.
@@ -24,4 +28,16 @@ class Platform(object):
"""Clean up platform data."""
pass
+ def _generate_ssh_keys(self, data_dir):
+ """Generate SSH keys to be used with image."""
+ filename = os.path.join(data_dir, 'id_rsa')
+
+ if os.path.exists(filename):
+ c_util.del_file(filename)
+
+ c_util.subp(['ssh-keygen', '-t', 'rsa', '-b', '4096',
+ '-f', filename, '-P', '',
+ '-C', 'ubuntu@cloud_test'],
+ capture=True)
+
# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/releases.yaml b/tests/cloud_tests/releases.yaml
index e593380..f968c3b 100644
--- a/tests/cloud_tests/releases.yaml
+++ b/tests/cloud_tests/releases.yaml
@@ -27,6 +27,10 @@ default_release_config:
# features groups and additional feature settings
feature_groups: []
features: {}
+ ec2:
+ # Choose from: [ebs, instance-store]
+ root-store: ebs
+ boot_timeout: 300
nocloud-kvm:
mirror_url: https://cloud-images.ubuntu.com/daily
mirror_dir: '/srv/citest/nocloud-kvm'
diff --git a/tests/cloud_tests/setup_image.py b/tests/cloud_tests/setup_image.py
index 179f40d..6d24211 100644
--- a/tests/cloud_tests/setup_image.py
+++ b/tests/cloud_tests/setup_image.py
@@ -5,7 +5,6 @@
from functools import partial
import os
-from cloudinit import util as c_util
from tests.cloud_tests import LOG
from tests.cloud_tests import stage, util
@@ -192,20 +191,6 @@ def enable_repo(args, image):
image.execute(cmd, description=msg)
-def generate_ssh_keys(data_dir):
- """Generate SSH keys to be used with image."""
- LOG.info('generating SSH keys')
- filename = os.path.join(data_dir, 'id_rsa')
-
- if os.path.exists(filename):
- c_util.del_file(filename)
-
- c_util.subp(['ssh-keygen', '-t', 'rsa', '-b', '4096',
- '-f', filename, '-P', '',
- '-C', 'ubuntu@cloud_test'],
- capture=True)
-
-
def setup_image(args, image):
"""Set up image as specified in args.
@@ -239,9 +224,6 @@ def setup_image(args, image):
LOG.info('setting up %s', image)
res = stage.run_stage(
'set up for {}'.format(image), calls, continue_after_error=False)
- LOG.debug('after setup complete, installed cloud-init version is: %s',
- installed_package_version(image, 'cloud-init'))
- generate_ssh_keys(args.data_dir)
return res
# vi: ts=4 expandtab
diff --git a/tests/cloud_tests/testcases/modules/apt_configure_sources_list.py b/tests/cloud_tests/testcases/modules/apt_configure_sources_list.py
index 129d226..1c94aee 100644
--- a/tests/cloud_tests/testcases/modules/apt_configure_sources_list.py
+++ b/tests/cloud_tests/testcases/modules/apt_configure_sources_list.py
@@ -9,18 +9,15 @@ class TestAptconfigureSourcesList(base.CloudTestCase):
def test_sources_list(self):
"""Test sources.list includes sources."""
+ url_archive = r'http:\/\/.*archive\.ubuntu\.com\/ubuntu\/? [a-z].*'
+ url_security = r'http:\/\/.*security\.ubuntu\.com\/ubuntu\/? [a-z].*'
+
out = self.get_data_file('sources.list')
- self.assertRegex(out, r'deb http:\/\/archive.ubuntu.com\/ubuntu '
- '[a-z].* main restricted')
- self.assertRegex(out, r'deb-src http:\/\/archive.ubuntu.com\/ubuntu '
- '[a-z].* main restricted')
- self.assertRegex(out, r'deb http:\/\/archive.ubuntu.com\/ubuntu '
- '[a-z].* universe restricted')
- self.assertRegex(out, r'deb-src http:\/\/archive.ubuntu.com\/ubuntu '
- '[a-z].* universe restricted')
- self.assertRegex(out, r'deb http:\/\/security.ubuntu.com\/ubuntu '
- '[a-z].*security multiverse')
- self.assertRegex(out, r'deb-src http:\/\/security.ubuntu.com\/ubuntu '
- '[a-z].*security multiverse')
+ self.assertRegex(out, r'deb %s main restricted' % url_archive)
+ self.assertRegex(out, r'deb-src %s main restricted' % url_archive)
+ self.assertRegex(out, r'deb %s universe restricted' % url_archive)
+ self.assertRegex(out, r'deb-src %s universe restricted' % url_archive)
+ self.assertRegex(out, r'deb %ssecurity multiverse' % url_security)
+ self.assertRegex(out, r'deb-src %ssecurity multiverse' % url_security)
# vi: ts=4 expandtab
diff --git a/tox.ini b/tox.ini
index 0802072..bc9e5c2 100644
--- a/tox.ini
+++ b/tox.ini
@@ -134,4 +134,5 @@ passenv = HOME
deps =
pylxd==2.2.4
paramiko==2.3.1
+ boto3==1.4.8
bzr+lp:simplestreams
Follow ups
-
[Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Joshua Powers, 2017-12-19
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Chad Smith, 2017-12-19
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Scott Moser, 2017-12-19
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Chad Smith, 2017-12-15
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Scott Moser, 2017-12-15
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Scott Moser, 2017-12-15
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Ryan Harper, 2017-12-15
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Joshua Powers, 2017-12-15
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Server Team CI bot, 2017-12-15
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Ryan Harper, 2017-12-14
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Ryan Harper, 2017-12-14
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Ryan Harper, 2017-12-14
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Scott Moser, 2017-12-14
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Server Team CI bot, 2017-12-14
-
Re: [Merge] ~powersj/cloud-init:cii-enable-ec2 into cloud-init:master
From: Server Team CI bot, 2017-12-14
