yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1686514@xxxxxxxxxxxxxxxxxx>]

This bug was fixed in the package cloud-init - 0.7.9-153-g16a7302f-
0ubuntu1~16.10.1

---------------
cloud-init (0.7.9-153-g16a7302f-0ubuntu1~16.10.1) yakkety-proposed; urgency=medium

  * New upstream snapshot.
    - net: fix reading and rendering addresses in cidr format.
      [Dimitri John Ledkov] (LP: #1689346, #1684349)
    - disk_setup: udev settle before attempting partitioning or fs creation.
      (LP: #1692093)
    - GCE: Update the attribute used to find instance SSH keys.
      [Daniel Watkins] (LP: #1693582)
    - nplan: For bonds, allow dashed or underscore names of keys.
      [Dimitri John Ledkov] (LP: #1690480)
    - tests: python2.6: fix unit tests usage of assertNone and format.
    - tests: update docstring on test_configured_list_with_none
    - fix tools/ds-identify to not write None twice.
    - tox/build: do not package depend on style requirements.
    - tests: ntp: Restructure cc_ntp unit tests. [Chad Smith]
    - flake8: move the pinned version of flake8 up to 3.3.0
    - tests: Apply workaround for snapd bug in test case. [Joshua Powers]
    - RHEL/CentOS: Fix dual stack IPv4/IPv6 configuration.  [Andreas Karis]
    - disk_setup: fix several issues with gpt disk partitions. (LP: #1692087)
    - function spelling & docstring update [Joshua Powers]
    - tests: Fix unittest bug in ntp tests. [Joshua Powers]
    - tox: move pylint target to 1.7.1
    - Fix get_interfaces_by_mac for empty macs (LP: #1692028)
    - DigitalOcean: remove routes except for the public interface.
      [Ben Howard] (LP: #1681531.)
    - netplan: pass macaddress, when specified, for vlans
      [Dimitri John Ledkov] (LP: #1690388)
    - doc: various improvements for the docs on cc_users_groups.
      [Felix Dreissig]
    - cc_ntp: write template before installing and add service restart
      [Ryan Harper] (LP: #1645644)
    - tests: fix cloudstack unit tests to avoid accessing
      /var/lib/NetworkManager [Lars Kellogg-Stedman]
    - tests: fix hardcoded path to mkfs.ext4 [Joshua Powers] (LP: #1691517)
    - Actually skip warnings when .skip file is present.
      [Chris Brinker] (LP: #1691551)
    - netplan: fix netplan render_network_state signature.
      [Dimitri John Ledkov] (LP: #1685944)
    - Azure: fix reformatting of ephemeral disks on resize to large types.
      (LP: #1686514)
    - make deb: Add devscripts dependency for make deb.
      Cleanup packages/bddeb. [Chad Smith] (LP: #1685935)
    - openstack: fix log message copy/paste typo in _get_url_settings
      [Lars Kellogg-Stedman]
    - unittests: fix unittests run on centos [Joshua Powers]
    - Improve detection of snappy to include os-release and kernel cmdline.
      (LP: #1689944)
    - Add address to config entry generated by _klibc_to_config_entry.
      [Julien Castets] (LP: #1691135)
    - sysconfig: Raise ValueError when multiple default gateways are present.
      [Chad Smith] (LP: #1687485)
    - FreeBSD: improvements and fixes for use on Azure
      [Hongjiang Zhang] (LP: #1636345)
    - Add unit tests for ds-identify, fix Ec2 bug found.
    - fs_setup: if cmd is specified, use shell interpretation.
      [Paul Meyer] (LP: #1687712)
    - doc: document network configuration defaults policy and formats.
      [Ryan Harper]
    - doc: Fix name of "uri" key in docs for "cc_apt_configure" module
      [Felix Dreissig]
    - tests: Enable artful in integration tests [Joshua Powers]

 -- Scott Moser <smoser@xxxxxxxxxx>  Fri, 26 May 2017 16:08:21 -0400

** Changed in: cloud-init (Ubuntu Yakkety)
       Status: Fix Committed => Fix Released

** Changed in: cloud-init (Ubuntu Xenial)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1686514

Title:
  Azure: cloud-init does not handle reformatting GPT partition ephemeral
  disks

Status in cloud-init:
  Fix Committed
Status in cloud-init package in Ubuntu:
  Fix Released
Status in cloud-init source package in Xenial:
  Fix Released
Status in cloud-init source package in Yakkety:
  Fix Released
Status in cloud-init source package in Zesty:
  Fix Released
Status in cloud-init source package in Artful:
  Fix Released

Bug description:
  === Begin SRU Template ===
  [Impact]
  On Azure, cloud-init handles re-formatting the ephemeral disk.
  The contents of the ephemeral disk for a system will be replaced with
  a stock ephemeral disk in the following scenarios:
   a.) first boot
   b.) after a resize.
   c.) after a VM has been migrated from one host to another.

  That ephemeral disk is either
   1. mbr partitioned with 1 ntfs partition
   2. gpt partitioned with 2 partitions, a msft reserved partition and a
      ntfs partition.  This scenario is newer, and only occurs on
      larger instance types that have large ephemeral disks.

  cloud-init previously did not handle '2' above.

  [Test Case]
  Generically this is re-creatable by:
   1.) launch an instance on Azure
   2.) resize it to a L32 or G5 size
   3.) check to see that the ephemeral disk (/dev/disk/cloud/azure_resource)
       has been formatted to ext4.

  It is more easily recreated for testing and verification by:

  1. launch instance on azure
  2. re-partition the ephemeral disk to look like a "clean" disk above
  3. remove old logs, reboot
     $ dir=logs-$(date +"%Y%m%d-%H%M%S");
     $ mkdir -p $dir; mv /var/log/cloud-init* $dir
  4. ssh back in, expect that this the disk has an ext4 filesystem on it.
     And that it is mounted on /mnt.

     $ grep reformattable= /var/log/cloud-init.log
     2017-05-12 15:14:57,125 - DataSourceAzure.py[DEBUG]: reformattable=False: partition 1 (/dev/sdb1) on device /dev/disk/cloud/azure_resource was not ntfs formatted

     Or, if it was formatted, you'll see something like:
     2017-05-12 15:17:47,021 - DataSourceAzure.py[DEBUG]: reformattable=True: partition 2 (/dev/sdb2) on device /dev/disk/cloud/azure_resource was ntfs formatted and had no important files. Safe for reformatting.

     $ grep /mnt /proc/mounts
     /dev/sdb1 /mnt ext4 rw,relatime,data=ordered 0 0

  
  [Regression Potential]
  The change makes cloud-init accept another situation when it decides
  to be reformat a disk.  Reformatting of a disk could result in loss of
  customer data if the decision to do so results in a false positive.

  The fix came with some fairly extensive unit tests (TestCanDevBeReformatted)
  on the 'can_dev_be_reformatted' method.

  [Other Info]
  Upstream commit at
    https://git.launchpad.net/cloud-init/commit/?id=31b6f1732

  === End SRU Template ===

  
  Some Azure instances such as L32 or G5 have very large ephemeral disks which are partitioned via GPT vs. smaller ephemeral disks that have dos disklabels.

  At first boot of an instance the ephemeral disk is prepared and
  formatted properly. But if the instance is deallocated and then
  reallocated (thus receiving a new ephemeral disk) then cloud-init does
  not handle reformatting GPT partition ephemeral disks properly.
  Therefore /mnt is never mounted again.

  Test cases:
   1. Deploy an L32(s) VM on Azure
   2. Log in and ensure that the ephemeral disk is formatted and mounted to /mnt
   3. Via the portal you can "Redeploy" the VM to a new Azure Host (or alternatively stop and deallocate the VM for some time, and then restart/reallocate the VM).

  Expected Results:
   - After reallocation we expect the ephemeral disk to be formatted and mounted to /mnt.

  Actual Results:
   - After reallocation /mnt is not mounted and there are errors in the cloud-init log.

  *This was tested on Ubuntu 16.04 - but may affect other releases.

  Note: This bug a regression from previous cloud-init releases. GPT
  support for Azure ephemeral disk handling was added to cloud-init via
  this bug: https://bugs.launchpad.net/ubuntu/+source/cloud-
  init/+bug/1422919.

  Related bugs:
   * bug 1691489: fstab entries written by cloud-config may not be mounted

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1686514/+subscriptions