group.of.nepali.translators team mailing list archive

[Julian Andres Klode <1667512@xxxxxxxxxxxxxxxxxx>]

** Also affects: initramfs-tools (Ubuntu Bionic)
   Importance: Undecided
       Status: New

** Also affects: initramfs-tools (Ubuntu Xenial)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of नेपाली
भाषा समायोजकहरुको समूह, which is subscribed to Xenial.
Matching subscriptions: Ubuntu 16.04 Bugs
https://bugs.launchpad.net/bugs/1667512

Title:
  [SRU] update-initramfs should sync only the filesystem containing the
  initramfs image

Status in initramfs-tools package in Ubuntu:
  Fix Released
Status in initramfs-tools source package in Xenial:
  New
Status in initramfs-tools source package in Bionic:
  New
Status in initramfs-tools package in Debian:
  Fix Released

Bug description:
  [Impact]
  If there is lots of activity on a different filesystem, high io/cache pressure, or a stale mount updating initramfs might hang because the update-initramfs script calls sync. This leaves the system unresponsive and will require the user to forcefully kill dpkg.

  [Test Case]
  One easy way to simulate this is by copying a huge file over nfs. Note: do not use localhost, it is too fast for this purpose, a network bridge over qemu is good enough.

  1. Setup a nfs server and export a directory with write permissions
  for the client

  2. Copy a big file over NFS (assumes it is mounted at /mnt in the client)
  $ dd if=/dev/zero of=/mnt/big_zeros bs=512 count=1M status=progress
  Note: adjust the "count" accordingly to how fast/slow your system is.

  3. Call sync and verify that it only finished after "dd" is done.

  4. Repeat step #2 and now call "sync -f /boot/<some-initrd-image>".
  Verify that it finished immediatly.

  [Regression Potential]
  Although syncfs(2) man page [2] states that it works just the same as sync - except being limited to a filesystem -, any actual differences in the implementation can affect how the initrd image file is synced to the disk, leading to unexpected behavior.

  [Other]
  If syncfs is unsupported then the 'sync' binary will ensure that a normal system-wide sync is called [1]. The proposed patch won't help in scenarios where the initrd file is located in a busy filesystem (eg. no separated /boot partition).

  The 'sync' and 'syncfs' specifications are pretty much similar and
  syncfs is expected to work the same as a normal sync regarding data
  integrity. Also altought the patch introduces the initrd file as an
  extra argument which would cause 'sync -f' to fail if the file does
  not exist, the update-initramfs script would fail earlier than that
  because that file is created by the 'mv' call a few lines above. Thus
  no regressions are expected.

  [References]
  [1] http://git.savannah.gnu.org/cgit/coreutils.git/commit/src/sync.c?id=8b2bf5295f353016d4f5e6a2317d55b6a8e7fd00
  [2] http://manpages.ubuntu.com/manpages/xenial/man2/syncfs.2.html

  [Original bug report]
  At least four users, including myself, are having an issue with update-initramfs hanging while updating ubuntu 16.04. The bug has been documented while attempting an update to multiple kernel versions ( 4.4.0-24, 4.4.0-62,  4.4.0-63). The bug causes any apt-get update or install to fail, and may also lead to an unbootable system.

  User #1 (me):
  $ uname -a
  Linux <computername> 4.4.0-62-generic #83-Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

  $ sudo apt-get upgrade
  Fetched 1,571 MB in 2min 9s (12.2 MB/s)
  Extracting templates from packages: 100%
  Preconfiguring packages ...
  (Reading database ... 344634 files and directories currently installed.)
  Preparing to unpack .../base-files_9.4ubuntu4.4_amd64.deb ...
  Unpacking base-files (9.4ubuntu4.4) over (9.4ubuntu4.3) ...
  Processing triggers for plymouth-theme-ubuntu-text (0.9.2-3ubuntu13.1) ...
  update-initramfs: deferring update (trigger activated)
  Processing triggers for man-db (2.7.5-1) ...
  Processing triggers for cracklib-runtime (2.9.2-1build2) ...
  Processing triggers for install-info (6.1.0.dfsg.1-5) ...
  Processing triggers for initramfs-tools (0.122ubuntu8.8) ...
  update-initramfs: Generating /boot/initrd.img-4.4.0-63-generic

  <hangs overnight> Uh oh! Let's try to troubleshoot.

  $ sudo killall dpkg
  $ sudo dpkg --configure -a
  Setting up initramfs-tools (0.122ubuntu8.8) ...
  update-initramfs: deferring update (trigger activated)
  Setting up base-files (9.4ubuntu4.4) ...
  Installing new version of config file /etc/issue ...
  Installing new version of config file /etc/issue.net ...
  Installing new version of config file /etc/lsb-release ...
  Setting up linux-image-4.4.0-63-generic (4.4.0-63.84) ...
  Running depmod.
  update-initramfs: deferring update (hook will be called later)
  initrd.img(/boot/initrd.img-4.4.0-63-generic
  ) points to /boot/initrd.img-4.4.0-63-generic
   (/boot/initrd.img-4.4.0-63-generic) -- doing nothing at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 491.
  vmlinuz(/boot/vmlinuz-4.4.0-63-generic
  ) points to /boot/vmlinuz-4.4.0-63-generic
   (/boot/vmlinuz-4.4.0-63-generic) -- doing nothing at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 491.
  Examining /etc/kernel/postinst.d.
  run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-63-generic /boot/vmlinuz-4.4.0-63-generic
  run-parts: executing /etc/kernel/postinst.d/dkms 4.4.0-63-generic /boot/vmlinuz-4.4.0-63-generic
  run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-63-generic /boot/vmlinuz-4.4.0-63-generic
  update-initramfs: Generating /boot/initrd.img-4.4.0-63-generic
  ^CFailed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-63-generic.postinst line 1052.
  dpkg: error processing package linux-image-4.4.0-63-generic (--configure):
   subprocess installed post-installation script returned error exit status 2
  dpkg: dependency problems prevent configuration of linux-image-generic:
   linux-image-generic depends on linux-image-4.4.0-63-generic; however:
    Package linux-image-4.4.0-63-generic is not configured yet.

  dpkg: error processing package linux-image-generic (--configure):
   dependency problems - leaving unconfigured
  dpkg: dependency problems prevent configuration of linux-signed-image-4.4.0-63-generic:
   linux-signed-image-4.4.0-63-generic depends on linux-image-4.4.0-63-generic (= 4.4.0-63.84); however:
    Package linux-image-4.4.0-63-generic is not configured yet.

  dpkg: error processing package linux-signed-image-4.4.0-63-generic (--configure):
   dependency problems - leaving unconfigured
  dpkg: dependency problems prevent configuration of linux-signed-image-generic:
   linux-signed-image-generic depends on linux-signed-image-4.4.0-63-generic; however:
    Package linux-signed-image-4.4.0-63-generic is not configured yet.

  dpkg: error processing package linux-signed-image-generic (--configure):
   dependency problems - leaving unconfigured
  dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-63-generic:
   linux-image-extra-4.4.0-63-generic depends on linux-image-4.4.0-63-generic; however:
    Package linux-image-4.4.0-63-generic is not configured yet.

  dpkg: error processing package linux-image-extra-4.4.0-63-generic (--configure):
   dependency problems - leaving unconfigured
  dpkg: dependency problems prevent configuration of linux-generic:
   linux-generic depends on linux-image-generic (= 4.4.0.63.67); however:
    Package linux-image-generic is not configured yet.

  dpkg: error processing package linux-generic (--configure):
   dependency problems - leaving unconfigured
  dpkg: dependency problems prevent configuration of linux-signed-generic:
   linux-signed-generic depends on linux-signed-image-generic (= 4.4.0.63.67); however:
    Package linux-signed-image-generic is not configured yet.

  dpkg: error processing package linux-signed-generic (--configure):
   dependency problems - leaving unconfigured
  Processing triggers for initramfs-tools (0.122ubuntu8.8) ...
  update-initramfs: Generating /boot/initrd.img-4.4.0-63-generic
  ^Cdpkg: error processing package initramfs-tools (--configure):
   subprocess installed post-installation script was interrupted
  Processing triggers for plymouth-theme-ubuntu-text (0.9.2-3ubuntu13.1) ...
  update-initramfs: deferring update (trigger activated)
  Errors were encountered while processing:
   linux-image-4.4.0-63-generic
   linux-image-generic
   linux-signed-image-4.4.0-63-generic
   linux-signed-image-generic
   linux-image-extra-4.4.0-63-generic
   linux-generic
   linux-signed-generic
   initramfs-tools

  I verified that /boot has enough space:

  /dev/nvme0n1p2               496M  314M  157M  67% /boot

  Here's my /boot:

  $ ls -l /boot
  total 325378
  -rw-r--r-- 1 root root  1244118 Jan  6 16:44 abi-4.4.0-59-generic
  -rw-r--r-- 1 root root  1244118 Jan 18 07:59 abi-4.4.0-62-generic
  -rw-r--r-- 1 root root  1245512 Feb  1 11:39 abi-4.4.0-63-generic
  -rw-r--r-- 1 root root  1245512 Feb 20 05:40 abi-4.4.0-64-generic
  -rw-r--r-- 1 root root   190047 Jan  6 16:44 config-4.4.0-59-generic
  -rw-r--r-- 1 root root   190047 Jan 18 07:59 config-4.4.0-62-generic
  -rw-r--r-- 1 root root   190247 Feb  1 11:39 config-4.4.0-63-generic
  -rw-r--r-- 1 root root   190247 Feb 20 05:40 config-4.4.0-64-generic
  drwx------ 3 root root     4096 Dec 31  1969 efi
  drwxr-xr-x 5 root root     1024 Feb 23 13:23 grub
  -rw-r--r-- 1 root root 10931153 Feb 23 13:56 initrd.img-4.4.0-57-generic
  -rw-r--r-- 1 root root 46506399 Feb  9 05:37 initrd.img-4.4.0-59-generic
  -rw-r--r-- 2 root root 39526162 Feb 23 13:27 initrd.img-4.4.0-62-generic
  -rw-r--r-- 2 root root 39526162 Feb 23 13:27 initrd.img-4.4.0-62-generic.dpkg-bak
  -rw-r--r-- 1 root root        0 Feb 23 13:56 initrd.img-4.4.0-62-generic.new
  -rw-r--r-- 2 root root 26019417 Feb 23 13:11 initrd.img-4.4.0-63-generic
  -rw-r--r-- 2 root root 26019417 Feb 23 13:11 initrd.img-4.4.0-63-generic.dpkg-bak
  -rw-r--r-- 1 root root        0 Feb 23 13:56 initrd.img-4.4.0-63-generic.new
  -rw-r--r-- 1 root root 39516686 Feb 23 14:50 initrd.img-4.4.0-64-generic
  -rw-r--r-- 1 root root 39516505 Feb 23 13:33 initrd.img-4.4.0-64-generic.dpkg-bak
  drwx------ 2 root root    12288 Jul 31  2016 lost+found
  -rw-r--r-- 1 root root   182704 Jan 28  2016 memtest86+.bin
  -rw-r--r-- 1 root root   184380 Jan 28  2016 memtest86+.elf
  -rw-r--r-- 1 root root   184840 Jan 28  2016 memtest86+_multiboot.bin
  -rw------- 1 root root  3875594 Jan  6 16:44 System.map-4.4.0-59-generic
  -rw------- 1 root root  3875553 Jan 18 07:59 System.map-4.4.0-62-generic
  -rw------- 1 root root  3883990 Feb  1 11:39 System.map-4.4.0-63-generic
  -rw------- 1 root root  3883990 Feb 20 05:40 System.map-4.4.0-64-generic
  -rw------- 1 root root  7069136 Jan  6 16:44 vmlinuz-4.4.0-59-generic
  -rw------- 1 root root  7071064 Jan 11 07:08 vmlinuz-4.4.0-59-generic.efi.signed
  -rw------- 1 root root  7070992 Jan 18 07:59 vmlinuz-4.4.0-62-generic
  -rw------- 1 root root  7072920 Feb  3 08:04 vmlinuz-4.4.0-62-generic.efi.signed
  -rw------- 1 root root  7087088 Feb  1 11:39 vmlinuz-4.4.0-63-generic
  -rw------- 1 root root  7087152 Feb 20 05:40 vmlinuz-4.4.0-64-generic

  After uninstall of  linux-image-4.4.0-63-generic:

  $ sudo dpkg --status linux-image-4.4.0-63-genericPackage: linux-image-4.4.0-63-generic
  Status: deinstall ok config-files
  Priority: optional
  Section: kernel
  Installed-Size: 64800
  Maintainer: Ubuntu Kernel Team <kernel-team@xxxxxxxxxxxxxxxx>
  Architecture: amd64
  Source: linux
  Version: 4.4.0-63.84
  Provides: fuse-module, ivtv-modules, kvm-api-4, linux-image, redhat-cluster-modules, spl-dkms, virtualbox-guest-modules, zfs-dkms
  Depends: initramfs-tools | linux-initramfs-tool, kmod
  Recommends: grub-pc | grub-efi-amd64 | grub-efi-ia32 | grub | lilo
  Suggests: fdutils, linux-doc-4.4.0 | linux-source-4.4.0, linux-tools, linux-headers-4.4.0-63-generic

  There does seem to be a problem with linux-image-
  extra-4.4.0-63-generic. Trying to uninstall this triggers update-
  initramfs.

  $ sudo dpkg --status linux-image-extra-4.4.0-63-generic
  Package: linux-image-extra-4.4.0-63-generic
  Status: purge ok half-installed
  Priority: optional
  Section: kernel
  Installed-Size: 148939
  Maintainer: Ubuntu Kernel Team <kernel-team@xxxxxxxxxxxxxxxx>
  Architecture: amd64
  Source: linux
  Version: 4.4.0-63.84
  Depends: linux-image-4.4.0-63-generic, crda | wireless-crda

  I've tried http://askubuntu.com/questions/650732/apt-fails-to-remove-
  partially-installed-kernel-and-cant-install-any-other-packa/651596
  with no luck. At this point I'm not sure if a partially installed
  kernel is the problem, and I've run out of troubleshooting insight.

  User #2:
  https://ubuntuforums.org/showthread.php?t=2351895

  User #3:
  https://ubuntuforums.org/showthread.php?t=2353606

  User #4 (original question):

  When upgrading (apt-get upgrade) on 16.04 (ubuntu-mate) I get a hang
  (left from 9:50am til 16:00 or thereabouts)

  I tracked down as far as the call to /etc/kernel/postinst.d/initramfs-
  tools and added a -v option, the last relevant lines on the terminal
  when re-running `apt-get upgrade` are as follows:

  ...
  Adding binary /bin/date
  Adding binary /sbin/hwclock
  Adding binary /sbin/dumpe2fs
  Calling hook dmsetup
  Calling hook reiserfsprogs
  Building cpio /boot/initrd.img-4.4.0-24-generic.new initramfs
  ^CFailed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-24-generic.postinst line 1052.
  dpkg: error processing package linux-image-4.4.0-24-generic (--configure):
   subprocess installed post-installation script returned error exit status 2
  dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-24-generic:
   linux-image-extra-4.4.0-24-generic depends on linux-image-4.4.0-24-generic; however:
    Package linux-image-4.4.0-24-generic is not configured yet.

  Checking with `ps fax` before the interrupt, I get this tree -
  indicating a hang on `sync` task

  sudo dpkg --configure -a
   \_ dpkg --configure -a
       \_ /usr/bin/perl /var/lib/dpkg/info/linux-image-4.4.0-24-generic.postinst configure
           \_ run-parts --verbose --exit-on-error --arg=4.4.0-24-generic --arg=/boot/vmlinuz-4.4.0-24-generic /etc/kernel/postinst.d
               \_ /bin/sh -e /etc/kernel/postinst.d/initramfs-tools 4.4.0-24-generic /boot/vmlinuz-4.4.0-24-generic
                   \_ /bin/sh /usr/sbin/update-initramfs -v -c -t -k 4.4.0-24-generic -b /boot
                       \_ sync

  I tried moving the resulting initramfs img file in /boot so a new one
  would be created - a new one is indeed created, hang still occurs on
  `sync`

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/1667512/+subscriptions