yahoo-eng-team team mailing list archive

[Ryan Harper <1672833@xxxxxxxxxxxxxxxxxx>]

I'm marking this bug as Invalid the test-case specified is working in
upstream cloud-init as well as existing Ubuntu instances with cloud-
init.  I believe this issue observed is an Amazon Linux specific issue
and should be handled in the Amazon Linux forum.  If you feel there
still remains an upstream or Ubuntu-specific cloud-init issue, please
re-open with further details.

** Changed in: cloud-init
       Status: Incomplete => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1672833

Title:
  AWS NVME SSD (i3 family) ephemeral storage fails to mount via fs_setup

Status in cloud-init:
  Invalid

Bug description:
  I launched an AWS `i3.large` instance and attempted to configure it's
  NVME SSD ephemeral storage via fs_setup like so:

  fs_setup:
    - label: nvme0n1
      filesystem: ext4
      extra_opts: ["-E", "nodiscard"]
      device: /dev/nvme0n1
      partition: auto
  mounts:
    - [ /dev/nvme0n1, "/media/nvme0n1", "ext4", "defaults,discard", "0", "2" ]

  I also attempted to manually add a full volume partition using
  disk_store, but that did not change the error output below.

  
  The above configuration failed with the following messages in /var/log/cloud-init.log, leading to the ephemeral storage device provided by the 'i3' family not being mounted at the requested path:

  Mar 14 15:25:53 cloud-init[2718]: helpers.py[DEBUG]: Running config-mounts using lock (<FileLock using file '/var/lib/cloud/instances/i-0305fb6a917397bf8/sem/config_mounts'>)
  Mar 14 15:25:53 cloud-init[2718]: cc_mounts.py[DEBUG]: Attempting to determine the real name of /dev/nvme0n1
  Mar 14 15:25:53 cloud-init[2718]: cc_mounts.py[DEBUG]: Ignoring nonexistent named mount /dev/nvme0n1
  Mar 14 15:25:53 cloud-init[2718]: cc_mounts.py[DEBUG]: changed /dev/nvme0n1 => None

  
  The block device for these instance types (i3 family) are automatically mapped to /dev/nvme0n1 [1] and is visible via `sudo fdisk -l` run from the machine after boot:

  Disk /dev/nvme0n1: 475.0 GB, 475000000000 bytes, 927734375 sectors
  Units = sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes


  However, it appears that for these instance types, while the device is
  automatically made visible, it is not added to the EC2 instance
  metadata:

  [root@ip-10-0-1-247 ec2-user]# curl http://169.254.169.254/latest/meta-data/block-device-mapping/
  ami
  ebs1

  
  ...which is what fs_setup and disk_setup use to get the list of valid device names to perform operations on:

  https://git.launchpad.net/cloud-
  init/tree/cloudinit/sources/DataSourceEc2.py#n164

  
  It appears that fs_setup and disk_setup only consider devices provided by the instance metadata, as evidenced by the code path here:

  https://git.launchpad.net/cloud-init/tree/cloudinit/config/cc_disk_setup.py#n144
  https://git.launchpad.net/cloud-init/tree/cloudinit/cloud.py#n84

  which is no longer a valid strategy for mounting ephemeral instance
  storage via cloud-config.

  
  Note that after instance launch I was able to manually create the partition table and a full volume primary partition (at /dev/nvme0n1p1), then create a filesystem via 'sudo mkfs.ext4 /dev/nvme0n1p1', then finally mount the correct disk at '/media/nvme0n1p1' via 'mount'. As a workaround to the issue above, I'll simply run these commands manually via the 'runcmd' option in cloud-config.

  
  **[1] The AWS documentation for the block device mappings states here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html that "[NVMe instance store volumes] These volumes are mapped automatically as /dev/nvme[0-7]n1. You do not need to specify the NVMe volumes supported by an instance type in a block device mapping." This means they do not follow the standard 'ephemeral<number>' virtual name mapping that other instances use.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1672833/+subscriptions