yahoo-eng-team team mailing list archive

[Scott Moser <smoser@xxxxxxxxxx>]

** Attachment added: "journalctl -o short-precise after"
   https://bugs.launchpad.net/cloud-init/+bug/1717477/+attachment/4950748/+files/journal-short-precise-after.txt

** Description changed:

  We're running several machines with
  
-   cloud-init_0.7.9-153-g16a7302f-0ubuntu1~16.04.2
+   cloud-init_0.7.9-153-g16a7302f-0ubuntu1~16.04.2
  
  without problems.
  
  Just upgraded all machines to
  
-   cloud-init_0.7.9-233-ge586fe35-0ubuntu1~16.04.1
+   cloud-init_0.7.9-233-ge586fe35-0ubuntu1~16.04.1
  
  and rebooted them all.
  
  All machines report ordering cycles in their dmesg, resulting in systemd breaking the
  loop by NOT starting some important services, e.g. mouting local filesystems:
- 
  
  Sep 14 15:43:52.487945 noname systemd[1]: networking.service: Found ordering cycle on networking.service/start
  Sep 14 15:43:52.487952 noname systemd[1]: networking.service: Found dependency on local-fs.target/start
  Sep 14 15:43:52.487960 noname systemd[1]: networking.service: Found dependency on home.mount/start
  Sep 14 15:43:52.487968 noname systemd[1]: networking.service: Found dependency on systemd-fsck@dev-disk-by\x2dlabel-Home.service/start
  Sep 14 15:43:52.487975 noname systemd[1]: networking.service: Found dependency on cloud-init.service/start
  Sep 14 15:43:52.487982 noname systemd[1]: networking.service: Found dependency on networking.service/start
  Sep 14 15:43:52.488297 noname systemd[1]: networking.service: Breaking ordering cycle by deleting job local-fs.target/start
  Sep 14 15:43:52.488306 noname systemd[1]: local-fs.target: Job local-fs.target/start deleted to break ordering cycle starting with networking.service/start
  
- 
  % cat /etc/fstab
  LABEL=cloudimg-rootfs	/	 ext4	defaults,discard	0 1
  LABEL=Home		/home	 xfs	defaults,logbufs=8	0 2
  
- 
- In this case /home isn't mounted as a result of systemd breaking the loop, resulting in services depending on /home not being started.
+ In this case /home isn't mounted as a result of systemd breaking the
+ loop, resulting in services depending on /home not being started.
  
  1. Tell us your cloud provider
  
  AWS
  
  2. dpkg-query -W -f='${Version}' cloud-init
  
  0.7.9-233-ge586fe35-0ubuntu1~16.04.1
  
  3. Any appropriate cloud-init configuration you can provide us
  
  Nothing special - worked with 0.7.9-153-g16a7302f-0ubuntu1~16.04.2 on
  all machines without hassle.
  
- 
  The problem is this change:
  
  diff -uaNr 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
  --- 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	1970-01-01 01:00:00.000000000 +0100
  +++ 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	2017-07-28 22:28:47.000000000 +0200
  @@ -0,0 +1,2 @@
  +[Unit]
  +After=cloud-init.service
  
- 
  WORKAROUND
  ==========
  
  I just did a
  
-   rm /lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
+   rm /lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
  
  on all machines and rebooted them: no more dependency loops reported,
  everything works again.
+ 
+ Related bugs:
+  * bug 1686514: Azure: cloud-init does not handle reformatting GPT partition ephemeral disks
+  * bug 1691489: fstab entries written by cloud-config may not be mounted

** Also affects: cloud-init (Ubuntu)
   Importance: Undecided
       Status: New

** Changed in: cloud-init (Ubuntu)
       Status: New => Confirmed

** Changed in: cloud-init (Ubuntu)
   Importance: Undecided => High

** Also affects: cloud-init (Ubuntu Artful)
   Importance: High
       Status: Confirmed

** Also affects: cloud-init (Ubuntu Xenial)
   Importance: Undecided
       Status: New

** Also affects: cloud-init (Ubuntu Zesty)
   Importance: Undecided
       Status: New

** Changed in: cloud-init (Ubuntu Xenial)
       Status: New => Confirmed

** Changed in: cloud-init (Ubuntu Zesty)
       Status: New => Confirmed

** Changed in: cloud-init (Ubuntu Xenial)
   Importance: Undecided => High

** Changed in: cloud-init (Ubuntu Zesty)
   Importance: Undecided => High

** Tags added: regression-released

** Tags removed: regression-released
** Tags added: regression-release

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1717477

Title:
  cloud-init generates ordering cycle via After=cloud-init in systemd-
  fsck

Status in cloud-init:
  Confirmed
Status in cloud-init package in Ubuntu:
  Confirmed
Status in cloud-init source package in Xenial:
  Confirmed
Status in cloud-init source package in Zesty:
  Confirmed
Status in cloud-init source package in Artful:
  Confirmed

Bug description:
  We're running several machines with

    cloud-init_0.7.9-153-g16a7302f-0ubuntu1~16.04.2

  without problems.

  Just upgraded all machines to

    cloud-init_0.7.9-233-ge586fe35-0ubuntu1~16.04.1

  and rebooted them all.

  All machines report ordering cycles in their dmesg, resulting in systemd breaking the
  loop by NOT starting some important services, e.g. mouting local filesystems:

  Sep 14 15:43:52.487945 noname systemd[1]: networking.service: Found ordering cycle on networking.service/start
  Sep 14 15:43:52.487952 noname systemd[1]: networking.service: Found dependency on local-fs.target/start
  Sep 14 15:43:52.487960 noname systemd[1]: networking.service: Found dependency on home.mount/start
  Sep 14 15:43:52.487968 noname systemd[1]: networking.service: Found dependency on systemd-fsck@dev-disk-by\x2dlabel-Home.service/start
  Sep 14 15:43:52.487975 noname systemd[1]: networking.service: Found dependency on cloud-init.service/start
  Sep 14 15:43:52.487982 noname systemd[1]: networking.service: Found dependency on networking.service/start
  Sep 14 15:43:52.488297 noname systemd[1]: networking.service: Breaking ordering cycle by deleting job local-fs.target/start
  Sep 14 15:43:52.488306 noname systemd[1]: local-fs.target: Job local-fs.target/start deleted to break ordering cycle starting with networking.service/start

  % cat /etc/fstab
  LABEL=cloudimg-rootfs	/	 ext4	defaults,discard	0 1
  LABEL=Home		/home	 xfs	defaults,logbufs=8	0 2

  In this case /home isn't mounted as a result of systemd breaking the
  loop, resulting in services depending on /home not being started.

  1. Tell us your cloud provider

  AWS

  2. dpkg-query -W -f='${Version}' cloud-init

  0.7.9-233-ge586fe35-0ubuntu1~16.04.1

  3. Any appropriate cloud-init configuration you can provide us

  Nothing special - worked with 0.7.9-153-g16a7302f-0ubuntu1~16.04.2 on
  all machines without hassle.

  The problem is this change:

  diff -uaNr 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
  --- 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	1970-01-01 01:00:00.000000000 +0100
  +++ 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	2017-07-28 22:28:47.000000000 +0200
  @@ -0,0 +1,2 @@
  +[Unit]
  +After=cloud-init.service

  WORKAROUND
  ==========

  I just did a

    rm /lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf

  on all machines and rebooted them: no more dependency loops reported,
  everything works again.

  Related bugs:
   * bug 1686514: Azure: cloud-init does not handle reformatting GPT partition ephemeral disks
   * bug 1691489: fstab entries written by cloud-config may not be mounted

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1717477/+subscriptions