yahoo-eng-team team mailing list archive

Thread
Date

[Bug 1565638] [NEW] Cloud-init on Xenial won't deploy compressed content in "write_files" - says DecompressionError: Not a gzipped file

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Oded Arbel <oded@xxxxxxxxxx>
Date: Mon, 04 Apr 2016 07:10:29 -0000
Reply-to: Bug 1565638 <1565638@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx

Public bug reported:

I'm trying to deploy an Amazon EC2 instance using cloud-init.

When I'm compressing the files in the write_files section, the cloud-
init process doesn't write the files, and I get the following error in
the cloud-init.log :

Apr  3 16:27:16 ubuntu [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/config-write-files: FAIL: running config-write-files with frequency once-per-instance
Apr  3 16:27:16 ubuntu [CLOUDINIT] util.py[WARNING]: Running module write-files (<module 'cloudinit.config.cc_write_files' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py'>)
 failed
Apr  3 16:27:16 ubuntu [CLOUDINIT] util.py[DEBUG]: Running module write-files (<module 'cloudinit.config.cc_write_files' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py'>) failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 393, in decomp_gzip
    return decode_binary(gh.read())
  File "/usr/lib/python3.5/gzip.py", line 274, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.5/gzip.py", line 461, in read
    if not self._read_gzip_header():
  File "/usr/lib/python3.5/gzip.py", line 409, in _read_gzip_header
    raise OSError('Not a gzipped file (%r)' % magic)
OSError: Not a gzipped file (b"b'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 735, in _run_modules
    freq=freq)
  File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 70, in run
    return self._runners.run(name, functor, args, freq, clear_on_fail)
  File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 199, in run
    results = functor(*args)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py", line 39, in handle
    write_files(name, files, log)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py", line 74, in write_files
    contents = extract_contents(f_info.get('content', ''), extractions)
  File "/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py", line 98, in extract_contents
    result = util.decomp_gzip(result, quiet=False)
  File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 400, in decomp_gzip
    raise DecompressionError(six.text_type(e))
cloudinit.util.DecompressionError: Not a gzipped file (b"b'")

I've verified that the cloud init I'm submitting does encode the files
correctly by running this ruby code on the cloud-init YAML file:

> File.write("test.gz",
YAML.load(File.read("test.init"))["write_files"].first["content"])

Then:

$ file test.gz
test.gz: gzip compressed data, last modified: Mon Apr  4 06:55:53 2016, max compression, from Unix
$ python
Python 2.7.11+ (default, Mar 30 2016, 21:00:42) 
[GCC 5.3.1 20160330] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import gzip
>>> with gzip.open('test.gz', 'rb') as f:
...     file_content = f.read()
...     print file_content
... 
<TEXT CONTENT THAT I EXPECTED>

I've never implemented this with trusty, so I'm not sure how cloud-init
on trusty handles that.

** Affects: cloud-init
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1565638

Title:
  Cloud-init on Xenial won't deploy compressed content in "write_files"
  - says DecompressionError: Not a gzipped file

Status in cloud-init:
  New

Bug description:
  I'm trying to deploy an Amazon EC2 instance using cloud-init.

  When I'm compressing the files in the write_files section, the cloud-
  init process doesn't write the files, and I get the following error in
  the cloud-init.log :

  Apr  3 16:27:16 ubuntu [CLOUDINIT] handlers.py[DEBUG]: finish: init-network/config-write-files: FAIL: running config-write-files with frequency once-per-instance
  Apr  3 16:27:16 ubuntu [CLOUDINIT] util.py[WARNING]: Running module write-files (<module 'cloudinit.config.cc_write_files' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py'>)
   failed
  Apr  3 16:27:16 ubuntu [CLOUDINIT] util.py[DEBUG]: Running module write-files (<module 'cloudinit.config.cc_write_files' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py'>) failed
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 393, in decomp_gzip
      return decode_binary(gh.read())
    File "/usr/lib/python3.5/gzip.py", line 274, in read
      return self._buffer.read(size)
    File "/usr/lib/python3.5/gzip.py", line 461, in read
      if not self._read_gzip_header():
    File "/usr/lib/python3.5/gzip.py", line 409, in _read_gzip_header
      raise OSError('Not a gzipped file (%r)' % magic)
  OSError: Not a gzipped file (b"b'")

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 735, in _run_modules
      freq=freq)
    File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 70, in run
      return self._runners.run(name, functor, args, freq, clear_on_fail)
    File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 199, in run
      results = functor(*args)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py", line 39, in handle
      write_files(name, files, log)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py", line 74, in write_files
      contents = extract_contents(f_info.get('content', ''), extractions)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_write_files.py", line 98, in extract_contents
      result = util.decomp_gzip(result, quiet=False)
    File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 400, in decomp_gzip
      raise DecompressionError(six.text_type(e))
  cloudinit.util.DecompressionError: Not a gzipped file (b"b'")

  I've verified that the cloud init I'm submitting does encode the files
  correctly by running this ruby code on the cloud-init YAML file:

  > File.write("test.gz",
  YAML.load(File.read("test.init"))["write_files"].first["content"])

  Then:

  $ file test.gz
  test.gz: gzip compressed data, last modified: Mon Apr  4 06:55:53 2016, max compression, from Unix
  $ python
  Python 2.7.11+ (default, Mar 30 2016, 21:00:42) 
  [GCC 5.3.1 20160330] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import gzip
  >>> with gzip.open('test.gz', 'rb') as f:
  ...     file_content = f.read()
  ...     print file_content
  ... 
  <TEXT CONTENT THAT I EXPECTED>

  I've never implemented this with trusty, so I'm not sure how cloud-
  init on trusty handles that.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1565638/+subscriptions

Follow ups

[Bug 1565638] Re: Cloud-init on Xenial won't deploy compressed content in "write_files" - says DecompressionError: Not a gzipped file
From: Scott Moser, 2016-08-10
[Bug 1565638] Re: Cloud-init on Xenial won't deploy compressed content in "write_files" - says DecompressionError: Not a gzipped file
From: Launchpad Bug Tracker, 2016-04-06
[Bug 1565638] Re: Cloud-init on Xenial won't deploy compressed content in "write_files" - says DecompressionError: Not a gzipped file
From: Scott Moser, 2016-04-04