yahoo-eng-team team mailing list archive

[thermoman <1717477@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

We're running several machines with

  cloud-init_0.7.9-153-g16a7302f-0ubuntu1~16.04.2

without problems.

Just upgraded all machines to

  cloud-init_0.7.9-233-ge586fe35-0ubuntu1~16.04.1

and rebooted them all.

All machines report ordering cycles in their dmesg, resulting in systemd breaking the
loop by NOT starting some important services, e.g. mouting local filesystems:


Sep 14 15:43:52.487945 noname systemd[1]: networking.service: Found ordering cycle on networking.service/start
Sep 14 15:43:52.487952 noname systemd[1]: networking.service: Found dependency on local-fs.target/start
Sep 14 15:43:52.487960 noname systemd[1]: networking.service: Found dependency on home.mount/start
Sep 14 15:43:52.487968 noname systemd[1]: networking.service: Found dependency on systemd-fsck@dev-disk-by\x2dlabel-Home.service/start
Sep 14 15:43:52.487975 noname systemd[1]: networking.service: Found dependency on cloud-init.service/start
Sep 14 15:43:52.487982 noname systemd[1]: networking.service: Found dependency on networking.service/start
Sep 14 15:43:52.488297 noname systemd[1]: networking.service: Breaking ordering cycle by deleting job local-fs.target/start
Sep 14 15:43:52.488306 noname systemd[1]: local-fs.target: Job local-fs.target/start deleted to break ordering cycle starting with networking.service/start


% cat /etc/fstab
LABEL=cloudimg-rootfs	/	 ext4	defaults,discard	0 1
LABEL=Home		/home	 xfs	defaults,logbufs=8	0 2


In this case /home isn't mounted as a result of systemd breaking the loop, resulting in services depending on /home not being started.

1. Tell us your cloud provider

AWS

2. dpkg-query -W -f='${Version}' cloud-init

0.7.9-233-ge586fe35-0ubuntu1~16.04.1

3. Any appropriate cloud-init configuration you can provide us

Nothing special - worked with 0.7.9-153-g16a7302f-0ubuntu1~16.04.2 on
all machines without hassle.


The problem is this change:

diff -uaNr 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
--- 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	1970-01-01 01:00:00.000000000 +0100
+++ 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	2017-07-28 22:28:47.000000000 +0200
@@ -0,0 +1,2 @@
+[Unit]
+After=cloud-init.service


WORKAROUND
==========

I just did a

  rm /lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf

on all machines and rebooted them: no more dependency loops reported,
everything works again.

** Affects: cloud-init
     Importance: Undecided
         Status: New


** Tags: systemd

** Tags added: cycle ordering

** Tags removed: cycle ordering
** Tags added: systemd

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1717477

Title:
  cloud-init generates ordering cycle via After=cloud-init in systemd-
  fsck

Status in cloud-init:
  New

Bug description:
  We're running several machines with

    cloud-init_0.7.9-153-g16a7302f-0ubuntu1~16.04.2

  without problems.

  Just upgraded all machines to

    cloud-init_0.7.9-233-ge586fe35-0ubuntu1~16.04.1

  and rebooted them all.

  All machines report ordering cycles in their dmesg, resulting in systemd breaking the
  loop by NOT starting some important services, e.g. mouting local filesystems:

  
  Sep 14 15:43:52.487945 noname systemd[1]: networking.service: Found ordering cycle on networking.service/start
  Sep 14 15:43:52.487952 noname systemd[1]: networking.service: Found dependency on local-fs.target/start
  Sep 14 15:43:52.487960 noname systemd[1]: networking.service: Found dependency on home.mount/start
  Sep 14 15:43:52.487968 noname systemd[1]: networking.service: Found dependency on systemd-fsck@dev-disk-by\x2dlabel-Home.service/start
  Sep 14 15:43:52.487975 noname systemd[1]: networking.service: Found dependency on cloud-init.service/start
  Sep 14 15:43:52.487982 noname systemd[1]: networking.service: Found dependency on networking.service/start
  Sep 14 15:43:52.488297 noname systemd[1]: networking.service: Breaking ordering cycle by deleting job local-fs.target/start
  Sep 14 15:43:52.488306 noname systemd[1]: local-fs.target: Job local-fs.target/start deleted to break ordering cycle starting with networking.service/start

  
  % cat /etc/fstab
  LABEL=cloudimg-rootfs	/	 ext4	defaults,discard	0 1
  LABEL=Home		/home	 xfs	defaults,logbufs=8	0 2

  
  In this case /home isn't mounted as a result of systemd breaking the loop, resulting in services depending on /home not being started.

  1. Tell us your cloud provider

  AWS

  2. dpkg-query -W -f='${Version}' cloud-init

  0.7.9-233-ge586fe35-0ubuntu1~16.04.1

  3. Any appropriate cloud-init configuration you can provide us

  Nothing special - worked with 0.7.9-153-g16a7302f-0ubuntu1~16.04.2 on
  all machines without hassle.

  
  The problem is this change:

  diff -uaNr 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
  --- 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	1970-01-01 01:00:00.000000000 +0100
  +++ 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf	2017-07-28 22:28:47.000000000 +0200
  @@ -0,0 +1,2 @@
  +[Unit]
  +After=cloud-init.service

  
  WORKAROUND
  ==========

  I just did a

    rm /lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf

  on all machines and rebooted them: no more dependency loops reported,
  everything works again.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1717477/+subscriptions