yahoo-eng-team team mailing list archive

[Mathew Hodson <1834875@xxxxxxxxxxxxxxxxxx>]

** Project changed: cloud-init => cloud-initramfs-tools

** Changed in: cloud-initramfs-tools
       Status: Incomplete => Fix Released

** Bug watch removed: github.com/systemd/systemd/issues #12258
   https://github.com/systemd/systemd/issues/12258

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1834875

Title:
  cloud-init growpart race with udev

Status in cloud-initramfs-tools:
  Fix Released
Status in cloud-utils:
  Fix Committed
Status in cloud-initramfs-tools package in Ubuntu:
  Fix Released
Status in cloud-utils package in Ubuntu:
  Fix Released
Status in cloud-initramfs-tools source package in Eoan:
  Fix Committed
Status in cloud-utils source package in Eoan:
  Fix Committed

Bug description:
  [impact]
  On Azure, it happens regularly (20-30%), that cloud-init's growpart module fails to extend the partition to full size.

  Such as in this example:

  ========================================

  2019-06-28 12:24:18,666 - util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
  2019-06-28 12:24:19,157 - util.py[DEBUG]: Running command ['growpart', '/dev/sda', '1'] with allowed return codes [0] (shell=False, capture=True)
  2019-06-28 12:24:19,726 - util.py[DEBUG]: resize_devices took 1.075 seconds
  2019-06-28 12:24:19,726 - handlers.py[DEBUG]: finish: init-network/config-growpart: FAIL: running config-growpart with frequency always
  2019-06-28 12:24:19,727 - util.py[WARNING]: Running module growpart (<module 'cloudinit.config.cc_growpart' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py'>) failed
  2019-06-28 12:24:19,727 - util.py[DEBUG]: Running module growpart (<module 'cloudinit.config.cc_growpart' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py'>) failed
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 812, in _run_modules
      freq=freq)
    File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 54, in run
      return self._runners.run(name, functor, args, freq, clear_on_fail)
    File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 187, in run
      results = functor(*args)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 351, in handle
      func=resize_devices, args=(resizer, devices))
    File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 2521, in log_time
      ret = func(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 298, in resize_devices
      (old, new) = resizer.resize(disk, ptnum, blockdev)
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 159, in resize
      return (before, get_size(partdev))
    File "/usr/lib/python3/dist-packages/cloudinit/config/cc_growpart.py", line 198, in get_size
      fd = os.open(filename, os.O_RDONLY)
  FileNotFoundError: [Errno 2] No such file or directory: '/dev/disk/by-partuuid/a5f2b49f-abd6-427f-bbc4-ba5559235cf3'

  ========================================

  @rcj suggested this is a race with udev. This seems to only happen on
  Cosmic and later.

  [test case]
  slightly unfortunately, the only way we've been able to reliably reproduce this is to get the CPC team to run their tests in azure, so we'll have to rely on them for SRU verification too

  [regression potential]
  The change is fairly simple and was thoroughly reviewed and tested before it got uploaded to focal, and the backport is trivial so I am confident that the regression potential is slight.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-initramfs-tools/+bug/1834875/+subscriptions