cloud-init-dev team mailing list archive

[Scott Moser <ssmoser2+ubuntu@xxxxxxxxx>]

Scott Moser has proposed merging ~smoser/cloud-init:cleanup/ec2-console-as-bytes into cloud-init:master.

Commit message:
tests: Fix EC2 Platform to return console output as bytes.

The EC2 test platform uses boto, and boto decodes console output
with decode('utf-8', 'replace).  It is known that Ubuntu consoles
contain non-utf8 characters, making this call lossy.

The change here is to patch the boto session to include a OutputBytes
entry in the console_output response, and then to utilize that in
console_log.

More information on problem and solution at:
   https://github.com/boto/botocore/issues/1351

Requested reviews:
  cloud-init commiters (cloud-init-dev)

For more details, see:
https://code.launchpad.net/~smoser/cloud-init/+git/cloud-init/+merge/336366

see commit message
-- 
Your team cloud-init commiters is requested to review the proposed merge of ~smoser/cloud-init:cleanup/ec2-console-as-bytes into cloud-init:master.

diff --git a/tests/cloud_tests/platforms/ec2/instance.py b/tests/cloud_tests/platforms/ec2/instance.py
index 4ba737a..8849b4b 100644
--- a/tests/cloud_tests/platforms/ec2/instance.py
+++ b/tests/cloud_tests/platforms/ec2/instance.py
@@ -46,9 +46,11 @@ class EC2Instance(Instance):
         may return empty string.
         """
         try:
-            return self.instance.console_output()['Output'].encode()
+            # OutputBytes comes from solution to
+            # https://github.com/boto/botocore/issues/1351
+            return self.instance.console_output()['OutputBytes']
         except KeyError:
-            return b''
+            return 'No Console Output [%s]' % self.instance).encode()
 
     def destroy(self):
         """Clean up instance."""
diff --git a/tests/cloud_tests/platforms/ec2/platform.py b/tests/cloud_tests/platforms/ec2/platform.py
index fdb17ba..ba246fe 100644
--- a/tests/cloud_tests/platforms/ec2/platform.py
+++ b/tests/cloud_tests/platforms/ec2/platform.py
@@ -6,6 +6,8 @@ import os
 
 import boto3
 import botocore
+from botocore import session, handlers
+import base64
 
 from ..platforms import Platform
 from .image import EC2Image
@@ -28,7 +30,7 @@ class EC2Platform(Platform):
         self.instance_type = config['instance-type']
 
         try:
-            self.ec2_client = boto3.client('ec2')
+            self.ec2_client = get_ec2_client()
             self.ec2_resource = boto3.resource('ec2')
             self.ec2_region = boto3.Session().region_name
             self.key_name = self._upload_public_key(config)
@@ -228,4 +230,28 @@ class EC2Platform(Platform):
 
         return self.tag
 
+
+def _decode_console_output_as_bytes(parsed, **kwargs):
+    """Provide console output as bytes in OutputBytes.
+
+       For this to be useful, the session has to have had the
+       decode_console_output handler unregistered already.
+
+       https://github.com/boto/botocore/issues/1351 ."""
+    if 'Output' not in parsed:
+        return
+    orig = parsed['Output']
+    handlers.decode_console_output(parsed, **kwargs)
+    parsed['OutputBytes'] = base64.b64decode(orig)
+
+
+def get_ec2_client():
+    mysess = session.get_session()
+    mysess.unregister('after-call.ec2.GetConsoleOutput',
+                      handlers.decode_console_output)
+    mysess.register('after-call.ec2.GetConsoleOutput',
+                    _decode_console_output_as_bytes)
+    return mysess.create_client('ec2')
+
+
 # vi: ts=4 expandtab