cloud-init-dev team mailing list archive
-
cloud-init-dev team
-
Mailing list archive
-
Message #06366
[Merge] ~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master
Chris Glass has proposed merging ~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master.
Commit message:
New data source for the Exoscale.com cloud platform
- dsidentify switches to the new Exoscale datasource on matching DMI name
- New Exoscale datasource added
Requested reviews:
cloud-init commiters (cloud-init-dev)
For more details, see:
https://code.launchpad.net/~tribaal/cloud-init/+git/cloud-init/+merge/369516
New data source for the Exoscale.com cloud platform
While the Exoscale cloud platform started as a Cloudstack fork, over the years
more and more of the codebase was rewritten, replaced, or otherwise
fundamentally changed.
Our recent feature development around networking made it clear that while we
intend for our cloud platform to remain as compatible with the Cloudstack
datasource in the forseeable future (once forced - see below), it is time for
us to bite the proverbial bullet and submit our own datasource for review.
Networking glitch:
The Cloudstack datasource doesn't work for us in the scenario where a user boots
a VM attached to the default network and one or more private network(s) running
DHCP (hosted by us, or customer-deployed): in some race condition the private
interface will be the lease on which the datasource will try to query for
metadata - which can't work. The submitted datasource fixes the problem by
forcing the datasource IP address to always be the same, so that we don't need
to depend on DHCP at all to determine it, fixing the race (we ensure routes to
that address are provided).
DMI product name change:
Along with this change, we intend to switch the DMI product name from
"CloudStack KVM Hypervisor" to "Exoscale Compute Platform" and the DMI
manufacturer string from "Apache Software Foundation" to "Exoscale".
Current templates situation:
We currently do not advertise custom templates to our customers, but intend to do
so more widely in the near future.
All of our current production templates force the Cloudstack datasource usage
(via configuration in /etc/cloud*).
Once this code lands, and our DMI product name change is performed, we will
document the requirements for (future) custom templates to be either:
- A recent enough cloud-init (containing the proposed code)
- Forcing the datasource to be "Cloudstack" for older cloud-init versions. This
will trigger the above-mentioned race, but that is a specific, relatively rare
use-case.
- Certain (future) features might only be available to customers running a
recent-enough cloud-init version.
Testing this code:
Inclusion in the cloud-init test suite in addition to the submitted unit tests
would be welcome.
--
Your team cloud-init commiters is requested to review the proposed merge of ~tribaal/cloud-init:feat/datasource-exoscale into cloud-init:master.
diff --git a/cloudinit/apport.py b/cloudinit/apport.py
index 22cb7fd..003ff1f 100644
--- a/cloudinit/apport.py
+++ b/cloudinit/apport.py
@@ -23,6 +23,7 @@ KNOWN_CLOUD_NAMES = [
'CloudStack',
'DigitalOcean',
'GCE - Google Compute Engine',
+ 'Exoscale',
'Hetzner Cloud',
'IBM - (aka SoftLayer or BlueMix)',
'LXD',
diff --git a/cloudinit/settings.py b/cloudinit/settings.py
index b1ebaad..2060d81 100644
--- a/cloudinit/settings.py
+++ b/cloudinit/settings.py
@@ -39,6 +39,7 @@ CFG_BUILTIN = {
'Hetzner',
'IBMCloud',
'Oracle',
+ 'Exoscale',
# At the end to act as a 'catch' when none of the above work...
'None',
],
diff --git a/cloudinit/sources/DataSourceExoscale.py b/cloudinit/sources/DataSourceExoscale.py
new file mode 100644
index 0000000..4588c9d
--- /dev/null
+++ b/cloudinit/sources/DataSourceExoscale.py
@@ -0,0 +1,126 @@
+import time
+
+from cloudinit import ec2_utils as ec2
+from cloudinit import log as logging
+from cloudinit import sources
+from cloudinit import url_helper
+
+API_VERSION = "latest"
+LOG = logging.getLogger(__name__)
+SERVICE_ADDRESS = "http://169.254.169.254";
+
+
+class DataSourceExoscale(sources.DataSource):
+
+ dsname = 'Exoscale'
+ url_timeout = 10
+ url_retries = 6
+ url_max_wait = 60
+
+ def __init__(self, sys_cfg, distro, paths):
+ sources.DataSource.__init__(self, sys_cfg, distro, paths)
+ LOG.info("Initializing the Exoscale datasource")
+ self.extra_config = {}
+
+ def get_password(self):
+ """Return the VM's passwords."""
+ LOG.info("Fetching password from metadata service")
+ password_url = "{}:8080".format(SERVICE_ADDRESS)
+ response = url_helper.read_file_or_url(
+ password_url,
+ ssl_details=None,
+ headers={"DomU_Request": "send_my_password"},
+ timeout=self.url_timeout,
+ retries=self.url_retries)
+ password = response.contents.decode('utf-8')
+ # the password is empty or already saved
+ if password in ['', 'saved_password']:
+ LOG.info("Password is missing or already saved")
+ return None
+ LOG.info("Found the password in metadata service")
+ # save the password
+ url_helper.read_file_or_url(
+ password_url,
+ ssl_details=None,
+ headers={"DomU_Request": "saved_password"},
+ timeout=self.url_timeout,
+ retries=self.url_retries)
+ LOG.info("password saved")
+ return password
+
+ def wait_for_metadata_service(self):
+ """Wait for the metadata service to be reachable."""
+ LOG.info("waiting for the metadata service")
+ start_time = time.time()
+
+ metadata_url = "{}/{}/meta-data/instance-id".format(
+ SERVICE_ADDRESS,
+ API_VERSION)
+
+ start_time = time.time()
+ url = url_helper.wait_for_url(
+ urls=[metadata_url],
+ max_wait=self.url_max_wait,
+ timeout=self.url_timeout,
+ status_cb=LOG.critical)
+
+ if url:
+ LOG.info("metadata service ok")
+ return True
+ else:
+ wait_time = int(time.time() - start_time)
+ LOG.critical(("Giving up on waiting for the metadata from %s"
+ " after %s seconds"),
+ url,
+ wait_time)
+ return False
+
+ def _get_data(self):
+ """Fetch the user data, the metadata and the VM password
+ from the metadata service."""
+ LOG.info("fetching data")
+ if not self.wait_for_metadata_service():
+ return False
+ start_time = time.time()
+ self.userdata_raw = ec2.get_instance_userdata(API_VERSION,
+ SERVICE_ADDRESS,
+ timeout=self.url_timeout,
+ retries=self.url_retries)
+ self.metadata = ec2.get_instance_metadata(API_VERSION,
+ SERVICE_ADDRESS,
+ timeout=self.url_timeout,
+ retries=self.url_retries)
+ password = self.get_password()
+ if password:
+ self.extra_config = {
+ 'ssh_pwauth': True,
+ 'password': password,
+ 'chpasswd': {
+ 'expire': False,
+ },
+ }
+ get_data_time = int(time.time() - start_time)
+ LOG.info("finished fetching the metadata in %s seconds",
+ get_data_time)
+ return True
+
+ def get_config_obj(self):
+ return self.extra_config
+
+ def get_instance_id(self):
+ return self.metadata['instance-id']
+
+ @property
+ def availability_zone(self):
+ return self.metadata['availability-zone']
+
+
+# Used to match classes to dependencies
+datasources = [
+ (DataSourceExoscale, (sources.DEP_FILESYSTEM, sources.DEP_NETWORK)),
+]
+
+
+# Return a list of data sources that match this set of dependencies
+def get_datasource_list(depends):
+ return sources.list_from_depends(depends, datasources)
diff --git a/tests/unittests/test_datasource/test_common.py b/tests/unittests/test_datasource/test_common.py
index 6b01a4e..24b0fac 100644
--- a/tests/unittests/test_datasource/test_common.py
+++ b/tests/unittests/test_datasource/test_common.py
@@ -13,6 +13,7 @@ from cloudinit.sources import (
DataSourceConfigDrive as ConfigDrive,
DataSourceDigitalOcean as DigitalOcean,
DataSourceEc2 as Ec2,
+ DataSourceExoscale as Exoscale,
DataSourceGCE as GCE,
DataSourceHetzner as Hetzner,
DataSourceIBMCloud as IBMCloud,
@@ -53,6 +54,7 @@ DEFAULT_NETWORK = [
CloudStack.DataSourceCloudStack,
DSNone.DataSourceNone,
Ec2.DataSourceEc2,
+ Exoscale.DataSourceExoscale,
GCE.DataSourceGCE,
MAAS.DataSourceMAAS,
NoCloud.DataSourceNoCloudNet,
diff --git a/tests/unittests/test_datasource/test_exoscale.py b/tests/unittests/test_datasource/test_exoscale.py
new file mode 100644
index 0000000..4bb9379
--- /dev/null
+++ b/tests/unittests/test_datasource/test_exoscale.py
@@ -0,0 +1,93 @@
+# This file is part of cloud-init. See LICENSE file for license information.
+from cloudinit import helpers
+from cloudinit.sources.DataSourceExoscale import (
+ API_VERSION,
+ DataSourceExoscale,
+ SERVICE_ADDRESS)
+from cloudinit.tests.helpers import HttprettyTestCase
+
+import httpretty
+
+
+@httpretty.activate
+class TestDatasourceExoscale(HttprettyTestCase):
+
+ def setUp(self):
+ super(TestDatasourceExoscale, self).setUp()
+ self.tmp = self.tmp_dir()
+
+ self.password_url = "{}:8080/".format(SERVICE_ADDRESS)
+ self.metadata_url = "{}/{}/meta-data/".format(SERVICE_ADDRESS,
+ API_VERSION)
+ self.userdata_url = "{}/{}/user-data".format(SERVICE_ADDRESS,
+ API_VERSION)
+
+ def test_password_saved(self):
+ """The password is not set when it is not found
+ in the metadata service."""
+ path = helpers.Paths({'run_dir': self.tmp})
+ ds = DataSourceExoscale({}, None, path)
+ httpretty.register_uri(httpretty.GET,
+ self.password_url,
+ body="saved_password")
+ self.assertFalse(ds.get_password())
+
+ def test_password_empty(self):
+ """No password is set if the metadata service returns
+ an empty string."""
+ path = helpers.Paths({'run_dir': self.tmp})
+ ds = DataSourceExoscale({}, None, path)
+ httpretty.register_uri(httpretty.GET,
+ self.password_url,
+ body="")
+ self.assertFalse(ds.get_password())
+
+ def test_password(self):
+ """The password is set to what is found in the metadata
+ service."""
+ path = helpers.Paths({'run_dir': self.tmp})
+ ds = DataSourceExoscale({}, None, path)
+ expected_password = "p@ssw0rd"
+ httpretty.register_uri(httpretty.GET,
+ self.password_url,
+ body=expected_password)
+ password = ds.get_password()
+ self.assertEqual(expected_password, password)
+
+ def test_get_data(self):
+ """The datasource conforms to expected behavior when supplied
+ full test data."""
+ path = helpers.Paths({'run_dir': self.tmp})
+ ds = DataSourceExoscale({}, None, path)
+ expected_password = "p@ssw0rd"
+ expected_id = "12345"
+ expected_hostname = "myname"
+ expected_userdata = "#cloud-config"
+ httpretty.register_uri(httpretty.GET,
+ self.userdata_url,
+ body=expected_userdata)
+ httpretty.register_uri(httpretty.GET,
+ self.password_url,
+ body=expected_password)
+ httpretty.register_uri(httpretty.GET,
+ self.metadata_url,
+ body="instance-id\nlocal-hostname")
+ httpretty.register_uri(httpretty.GET,
+ "{}local-hostname".format(self.metadata_url),
+ body=expected_hostname)
+ httpretty.register_uri(httpretty.GET,
+ "{}local-hostname".format(self.metadata_url),
+ body=expected_hostname)
+ httpretty.register_uri(httpretty.GET,
+ "{}instance-id".format(self.metadata_url),
+ body=expected_id)
+ ds._get_data()
+ self.assertEqual(ds.userdata_raw.decode("utf-8"), "#cloud-config")
+ self.assertEqual(ds.metadata, {"instance-id": expected_id,
+ "local-hostname": expected_hostname})
+ self.assertEqual(ds.get_config_obj(),
+ {'ssh_pwauth': True,
+ 'password': expected_password,
+ 'chpasswd': {
+ 'expire': False,
+ }})
diff --git a/tools/ds-identify b/tools/ds-identify
index e16708f..5727c24 100755
--- a/tools/ds-identify
+++ b/tools/ds-identify
@@ -124,7 +124,7 @@ DI_DSNAME=""
# be searched if there is no setting found in config.
DI_DSLIST_DEFAULT="MAAS ConfigDrive NoCloud AltCloud Azure Bigstep \
CloudSigma CloudStack DigitalOcean AliYun Ec2 GCE OpenNebula OpenStack \
-OVF SmartOS Scaleway Hetzner IBMCloud Oracle"
+OVF SmartOS Scaleway Hetzner IBMCloud Oracle Exoscale"
DI_DSLIST=""
DI_MODE=""
DI_ON_FOUND=""
@@ -553,6 +553,12 @@ dscheck_CloudStack() {
return $DS_NOT_FOUND
}
+dscheck_Exoscale() {
+ is_container && return ${DS_NOT_FOUND}
+ dmi_product_name_matches "Exoscale*" && return $DS_FOUND
+ return $DS_NOT_FOUND
+}
+
dscheck_CloudSigma() {
# http://paste.ubuntu.com/23624795/
dmi_product_name_matches "CloudSigma" && return $DS_FOUND
Follow ups
