debcrafters-packages team mailing list archive

[Jens Elkner <2081334@xxxxxxxxxxxxxxxxxx>]

Just upgraded a machine to noble (24.04) and found out, that dracut-
install is slow as hell. A "update-initramfs -c -k all" takes ~ 45s to
create a /boot/initrd.img-6.8.0-60-generic of ~16 MB (removed all other
kernels for testing). 4x longer than expected!!! If I comment out the
"/usr/lib/dracut/dracut-install ..." call in /usr/share/initramfs-
tools/hook-functions:_call_dracut_install(), it takes ~10s, only.
_call_dracut_install() gets called 15x, so /usr/lib/dracut/dracut-
install is probably the cause. On test the box was idle and uses 2x AMD
EPYC 9254  2.90 GHz 24-Core Processor with a NVMe SSD as root/boot disk
(SAMSUNG MZQL2960HCJR-00A07) and 1.5 TB RAM, all disks have > 500 GB
space available, so sufficient resources to get the job done pretty
fast. Not sure, why it behaves so bad, but calling about 62K readlink()
whereby ~50k of them return "-1 EINVAL" because the argument is a
directory might be a hint. E.g. 6203 fails for 'readlink("/sys",
0x7fffc447c2e0, 1023)', same for "/sys/devices" and
"/sys/devices/platform" as well as 6168 fails for "/sys/bus" and
"/sys/bus/platform" - so at least half of those calls seem to be total
overhead ... Not sure, whether this is the root cause for its terrible
slowness but fixing it would probably not hurt ;-)

-- 
You received this bug notification because you are a member of
Debcrafters packages, which is subscribed to initramfs-tools in Ubuntu.
https://bugs.launchpad.net/bugs/2081334

Title:
  further performance improvement for dracut-install 060

Status in initramfs-tools package in Ubuntu:
  Fix Released
Status in initramfs-tools source package in Noble:
  Fix Released

Bug description:
  [ Impact ]

  This is a follow-up for bug #2065180 implementing the suggestion from
  comment #44 there.

  When compared to Ubuntu 23.10, creating intramfs files with update-
  initramfs takes 2 to 5 times more time on ARM devices.

  IIUC, dracut-install usage was added to initramfs-tools to speed up
  the process. But now its way slower. Even running update-initramfs on
  jammy, which doesn't use dracut-install, is way faster then the time
  taken on Noble.

  first bad commit -
  https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f780390b874c870

  Updating the initrd on a Raspberry Pi Zero 2W on Ubuntu 24.04 (noble)
  with initramfs-tools 0.142ubuntu25.1 takes over six minutes:

  ```
  bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
  Benchmark 1: update-initramfs -u
    Time (mean ± σ): 402.751 s ± 5.592 s [User: 166.316 s, System: 228.909 s]
    Range (min … max): 394.380 s … 411.445 s 10 runs
  ```

  [ Test Plan ]

  1. Measure `update-initramfs -u` before the update.
  2. Log the content of the initrd before the update: `lsinitramfs /boot/initrd.img`
  3. update dracut-install / initramfs-tools-core
  4. Measure `update-initramfs -u`. It should be faster (the performance improvements on amd64 should be very small and might be within the measurement uncertainty).
  5. Check with lsinitramfs that the content of the newly generated initrd hasn't changed.

  [ Where problems could occur ]

  The code that is responsible for including the kernel modules into the
  initrd is touched. Negative consequences could be that some needed
  kernel modules will not be included any more (should be covered by the
  test case) or that building new initrds will fail.

  The initramfs-tools fix changes how manual_add_modules behaves.
  `manual_add_modules` does not copy kernel modules, but queues them for
  being copied when the newly added function `apply_add_modules` is
  called.

  I checked all instances of calls to `manual_add_modules` for possible
  regressions (see
  https://bugs.launchpad.net/ubuntu/+source/dracut/+bug/2065180/comments/15).
  Only miniramfs needs to be adjusted to also call `apply_add_modules`.
  But this change could break consumers of the `manual_add_modules`
  function that are outside of the Ubuntu archive. I googled for
  `apply_add_modules` but found no public outside users.

  [ Benchmarks ]

  Stock noble on a Raspberry Pi Zero 2W:

  ```
  bdrung@zero2w:~$ sudo hyperfine -r 5 "update-initramfs -u"
  Benchmark 1: update-initramfs -u
    Time (mean ± σ): 415.664 s ± 6.015 s [User: 166.728 s, System: 232.523 s]
    Range (min … max): 409.139 s … 422.632 s 5 runs
  ```

  noble with dracut-install 060+5-1ubuntu3.1 (with linux 6.8.0-1006.6 on
  2024-07-01):

  ```
  bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
  Benchmark 1: update-initramfs -u
    Time (mean ± σ): 248.054 s ± 5.569 s [User: 67.410 s, System: 169.412 s]
    Range (min … max): 238.909 s … 257.384 s 10 runs
  ```

  noble with dracut-install 060+5-1ubuntu3.2 (with linux 6.8.0-1007 on
  2024-07-11):

  ```
  bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
  Benchmark 1: update-initramfs -u
    Time (mean ± σ): 232.001 s ± 5.678 s [User: 55.456 s, System: 166.510 s]
    Range (min … max): 222.120 s … 239.610 s 10 runs
  ```

  noble with dracut-install 060+5-1ubuntu3.2 (with linux 6.8.0-1011.12
  and all updates on 2024-09-19):

  ```
  bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
    Time (mean ± σ): 219.439 s ± 4.869 s [User: 45.626 s, System: 163.172 s]
    Range (min … max): 213.340 s … 227.411 s 10 runs
  ```

  noble with initramfs-tools 0.142ubuntu25.2 (with dracut-install
  060+5-1ubuntu3.2, linux 6.8.0-1012.13, and all updates on 2024-09-30):

  ```
  bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
  Benchmark 1: update-initramfs -u
    Time (mean ± σ): 217.160 s ± 4.652 s [User: 45.608 s, System: 159.225 s]
    Range (min … max): 210.736 s … 225.802 s 10 runs
  ```

  noble with initramfs-tools 0.142ubuntu25.4 (with dracut-install
  060+5-1ubuntu3.2, linux 6.8.0-1012.13, and all updates on 2024-09-30):

  ```
  bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
  Benchmark 1: update-initramfs -u
    Time (mean ± σ): 204.143 s ± 4.441 s [User: 37.845 s, System: 153.927 s]
    Range (min … max): 195.729 s … 208.805 s 10 runs
  ```

  [ Reduce manual_add_modules calls ]

  Besides making the dracut-install calls faster, group the dracut-
  install calls. Since the fix in oracular can cause regressions in
  custom hooks that rely on the current behavior, the SRU takes a safe
  approach which includes following packages (stating how many dracut-
  install calls are used):

   * cryptsetup: 2 -> 1
   * lvm2: 8 -> 1
   * thin-provisioning-tools: 3 -> 1
   * open-iscsi: 9 -> 1
   * cloud-initramfs-tools: 5 -> 1

  dracut-install calls on a Raspberry Pi Zero 2W:

  | area | before | noble SRU | oracular |
  |--------------------------------------|--------|-----------|----------|
  | auto_add_modules + apply_add_modules | 8 | 5 | 5 |
  | calls by hooks + apply_add_modules | 42 | 20 | 2 |
  | hidden_dep_add_modules | 1 | 1 | 1 |
  | total | 51 | 26 | 8 |

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/initramfs-tools/+bug/2081334/+subscriptions