yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #67976
[Bug 1717477] Re: cloud-init generates ordering cycle via After=cloud-init in systemd-fsck
This bug is believed to be fixed in cloud-init in 17.1. If this is still
a problem for you, please make a comment and set the state back to New
Thank you.
** Changed in: cloud-init
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1717477
Title:
cloud-init generates ordering cycle via After=cloud-init in systemd-
fsck
Status in cloud-init:
Fix Released
Status in cloud-init package in Ubuntu:
Fix Released
Status in cloud-init source package in Xenial:
Fix Committed
Status in cloud-init source package in Zesty:
Fix Committed
Status in cloud-init source package in Artful:
Fix Released
Bug description:
http://pad.lv/1717477
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1717477
=== Begin SRU Template ===
[Impact]
Cloud-init's inclusion of a systemd drop-in file
/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
Caused a regression on systems that had entries in /etc/fstab
that were not authored by cloud-init (specifically that did not have
something like 'x-systemd.requires=cloud-init.service' in their
filesystem options.
[Test Case]
The test can be done on any cloud that has space to put a non-root
filesystem.
a.) launch instance
b.) upgrade to cloud-init to -updates pocket
c.) create a filesystem and put it in /etc/fstab
bdev="/dev/sdb1"
mkdir -p /mnt
mkfs.ext4 -F "$bdev"
echo "$bdev /mnt auto defaults 0 2" >> /etc/fstab
reboot
d.) see mention of 'ordering cycle' in journal
$ journalctl -o short-precise | grep -i ordering.cycle
Sep 15 14:08:48.331033 xenial-20170911-174122 systemd[1]: local-fs.target: Found ordering cycle on local-fs.target/start
Sep 15 14:08:48.331097 xenial-20170911-174122 systemd[1]: local-fs.target: Breaking ordering cycle by deleting job mnt.mount/start
Sep 15 14:08:48.331108 xenial-20170911-174122 systemd[1]: mnt.mount: Job mnt.mount/start deleted to break ordering cycle starting with local-fs.target/start
e.) upgrade to proposed
f.) reboot
g.) expect no mention of ordering cycle as seen in 'd'
$ journalctl -o short-precise | grep -i ordering.cycle || echo "no cycles"
no cycles
[Regression Potential]
This change will mean that bug 1691489 is present again.
That bug is much less severe and affects a much smaller set of users.
[Other Info]
Upstream commit at
https://git.launchpad.net/cloud-init/commit/?id=a2f8ce9c80
=== End SRU Template ===
We're running several machines with
cloud-init_0.7.9-153-g16a7302f-0ubuntu1~16.04.2
without problems.
Just upgraded all machines to
cloud-init_0.7.9-233-ge586fe35-0ubuntu1~16.04.1
and rebooted them all.
All machines report ordering cycles in their dmesg, resulting in systemd breaking the
loop by NOT starting some important services, e.g. mouting local filesystems:
Sep 14 15:43:52.487945 noname systemd[1]: networking.service: Found ordering cycle on networking.service/start
Sep 14 15:43:52.487952 noname systemd[1]: networking.service: Found dependency on local-fs.target/start
Sep 14 15:43:52.487960 noname systemd[1]: networking.service: Found dependency on home.mount/start
Sep 14 15:43:52.487968 noname systemd[1]: networking.service: Found dependency on systemd-fsck@dev-disk-by\x2dlabel-Home.service/start
Sep 14 15:43:52.487975 noname systemd[1]: networking.service: Found dependency on cloud-init.service/start
Sep 14 15:43:52.487982 noname systemd[1]: networking.service: Found dependency on networking.service/start
Sep 14 15:43:52.488297 noname systemd[1]: networking.service: Breaking ordering cycle by deleting job local-fs.target/start
Sep 14 15:43:52.488306 noname systemd[1]: local-fs.target: Job local-fs.target/start deleted to break ordering cycle starting with networking.service/start
% cat /etc/fstab
LABEL=cloudimg-rootfs / ext4 defaults,discard 0 1
LABEL=Home /home xfs defaults,logbufs=8 0 2
In this case /home isn't mounted as a result of systemd breaking the
loop, resulting in services depending on /home not being started.
1. Tell us your cloud provider
AWS
2. dpkg-query -W -f='${Version}' cloud-init
0.7.9-233-ge586fe35-0ubuntu1~16.04.1
3. Any appropriate cloud-init configuration you can provide us
Nothing special - worked with 0.7.9-153-g16a7302f-0ubuntu1~16.04.2 on
all machines without hassle.
The problem is this change:
diff -uaNr 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
--- 153/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf 1970-01-01 01:00:00.000000000 +0100
+++ 233/lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf 2017-07-28 22:28:47.000000000 +0200
@@ -0,0 +1,2 @@
+[Unit]
+After=cloud-init.service
WORKAROUND
==========
I just did a
rm /lib/systemd/system/systemd-fsck@.service.d/cloud-init.conf
on all machines and rebooted them: no more dependency loops reported,
everything works again.
Related bugs:
* bug 1686514: Azure: cloud-init does not handle reformatting GPT partition ephemeral disks
* bug 1691489: fstab entries written by cloud-config may not be mounted
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1717477/+subscriptions
References