yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1717477@xxxxxxxxxxxxxxxxxx>]

This bug was fixed in the package cloud-init -
0.7.9-233-ge586fe35-0ubuntu1~16.04.2

---------------
cloud-init (0.7.9-233-ge586fe35-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  * cherry-pick a2f8ce9c: Do not provide systemd-fsck drop-in which
    could cause systemd ordering loops (LP: #1717477).

 -- Scott Moser <smoser@xxxxxxxxxx>  Fri, 15 Sep 2017 15:23:38 -0400

** Changed in: cloud-init (Ubuntu Xenial)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1717477

Title:
  cloud-init generates ordering cycle via After=cloud-init in systemd-
  fsck

Status in cloud-init:
  Fix Released
Status in cloud-init package in Ubuntu:
  Fix Released
Status in cloud-init source package in Xenial:
  Fix Released
Status in cloud-init source package in Zesty:
  Fix Released
Status in cloud-init source package in Artful:
  Fix Released

Bug description:
  http://pad.lv/1717477
  https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1717477

  === Begin SRU Template ===
  [Impact]
  Cloud-init's inclusion of a systemd drop-in file
    /lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
  Caused a regression on systems that had entries in /etc/fstab
  that were not authored by cloud-init (specifically that did not have
  something like 'x-systemd.requires=cloud-init.service' in their
  filesystem options.

  [Test Case]
  The test can be done on any cloud that has space to put a non-root
  filesystem.

  a.) launch instance
  b.) upgrade to cloud-init to -updates pocket
  c.) create a filesystem and put it in /etc/fstab
      bdev="/dev/sdb1"
      mkdir -p /mnt
      mkfs.ext4 -F "$bdev"
      echo "$bdev /mnt auto defaults 0 2" >> /etc/fstab

      reboot
  d.) see mention of 'ordering cycle' in journal

      $ journalctl -o short-precise  | grep -i ordering.cycle
      Sep 15 14:08:48.331033 xenial-20170911-174122 systemd[1]: local-fs.target: Found ordering cycle on local-fs.target/start
      Sep 15 14:08:48.331097 xenial-20170911-174122 systemd[1]: local-fs.target: Breaking ordering cycle by deleting job mnt.mount/start
      Sep 15 14:08:48.331108 xenial-20170911-174122 systemd[1]: mnt.mount: Job mnt.mount/start deleted to break ordering cycle starting with local-fs.target/start

  e.) upgrade to proposed
  f.) reboot
  g.) expect no mention of ordering cycle as seen in 'd'
      $ journalctl -o short-precise  | grep -i ordering.cycle || echo "no cycles"
      no cycles

  [Regression Potential]
  This change will mean that bug 1691489 is present again.
  That bug is much less severe and affects a much smaller set of users.

  [Other Info]
  Upstream commit at
    https://git.launchpad.net/cloud-init/commit/?id=a2f8ce9c80

  === End SRU Template ===


  We're running several machines with

    cloud-init_0.7.9-153-g16a7302f-0ubuntu1~16.04.2

  without problems.

  Just upgraded all machines to

    cloud-init_0.7.9-233-ge586fe35-0ubuntu1~16.04.1

  and rebooted them all.

  All machines report ordering cycles in their dmesg, resulting in systemd breaking the
  loop by NOT starting some important services, e.g. mouting local filesystems:

  Sep 14 15:43:52.487945 noname systemd[1]: networking.service: Found ordering cycle on networking.service/start
  Sep 14 15:43:52.487952 noname systemd[1]: networking.service: Found dependency on local-fs.target/start
  Sep 14 15:43:52.487960 noname systemd[1]: networking.service: Found dependency on home.mount/start
  Sep 14 15:43:52.487968 noname systemd[1]: networking.service: Found dependency on systemd-fsck@dev-disk-by\x2dlabel-Home.service/start
  Sep 14 15:43:52.487975 noname systemd[1]: networking.service: Found dependency on cloud-init.service/start
  Sep 14 15:43:52.487982 noname systemd[1]: networking.service: Found dependency on networking.service/start
  Sep 14 15:43:52.488297 noname systemd[1]: networking.service: Breaking ordering cycle by deleting job local-fs.target/start
  Sep 14 15:43:52.488306 noname systemd[1]: local-fs.target: Job local-fs.target/start deleted to break ordering cycle starting with networking.service/start

  % cat /etc/fstab
  LABEL=cloudimg-rootfs	/	 ext4	defaults,discard	0 1
  LABEL=Home		/home	 xfs	defaults,logbufs=8	0 2

  In this case /home isn't mounted as a result of systemd breaking the
  loop, resulting in services depending on /home not being started.

  1. Tell us your cloud provider

  AWS

  2. dpkg-query -W -f='${Version}' cloud-init

  0.7.9-233-ge586fe35-0ubuntu1~16.04.1

  3. Any appropriate cloud-init configuration you can provide us

  Nothing special - worked with 0.7.9-153-g16a7302f-0ubuntu1~16.04.2 on
  all machines without hassle.

  The problem is this change:

  diff -uaNr 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
  --- 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	1970-01-01 01:00:00.000000000 +0100
  +++ 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	2017-07-28 22:28:47.000000000 +0200
  @@ -0,0 +1,2 @@
  +[Unit]
  +After=cloud-init.service

  WORKAROUND
  ==========

  I just did a

    rm /lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf

  on all machines and rebooted them: no more dependency loops reported,
  everything works again.

  Related bugs:
   * bug 1686514: Azure: cloud-init does not handle reformatting GPT partition ephemeral disks
   * bug 1691489: fstab entries written by cloud-config may not be mounted

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1717477/+subscriptions