canonical-ubuntu-qa team mailing list archive

Thread
Date
[Bug 1989258] Re: multiple kernel oops regarding hung tasks delaying boot

To: canonical-ubuntu-qa@xxxxxxxxxxxxxxxxxxx
From: Andrea Righi <1989258@xxxxxxxxxxxxxxxxxx>
Date: Tue, 13 Sep 2022 07:48:52 -0000
Reply-to: Bug 1989258 <1989258@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Hi Brian, we have also experienced this issue during our tests, but it's
really hard to reproduce it on our side.

It seems to be a regression in virtio-rng and I've prepared a test
kernel, but we still haven't found a way to reproduce it systematically,
so I couldn't verify if this test kernel is actually fixing the problem
or not.

Are you able to reproduce this issue easily? In that case, it'd be great if you could a quick test with this kernel:
https://kernel.ubuntu.com/~arighi/lp-1989258/

(I've uploaded just the debs here, but if you prefer a ppa let me know)

Thanks in advance!

-- 
You received this bug notification because you are a member of
Canonical's Ubuntu QA, which is subscribed to Auto Package Testing.
https://bugs.launchpad.net/bugs/1989258

Title:
  multiple kernel oops regarding hung tasks delaying boot

Status in Auto Package Testing:
  New
Status in linux package in Ubuntu:
  Confirmed

Bug description:
  Kinetic amd64 instances in the autopkgtest environment are regularly
  experiencing a slow boot process due to a hung task timeout which
  happens multiple times.  The oops appears below:

  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.485373] INFO: task systemd-udevd:140 blocked for more than 1208 seconds.
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.487120]       Not tainted 5.19.0-15-generic #15-Ubuntu
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.490936] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.492717] task:systemd-udevd   state:D stack:    0 pid:  140 ppid:   137 flags:0x00004006
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.494612] Call Trace:
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.495202]  <TASK>
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.495727]  __schedule+0x23b/0x5c0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.496559]  ? sched_clock_cpu+0x12/0xf0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.497482]  schedule+0x50/0xc0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.498237]  schedule_preempt_disabled+0x15/0x30
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.499306]  __mutex_lock.constprop.0+0x4e7/0x760
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.500409]  __mutex_lock_slowpath+0x13/0x20
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.501420]  mutex_lock+0x36/0x40
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.502211]  add_early_randomness+0x1b/0xc0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.503178]  hwrng_register+0x13e/0x1c0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.504068]  virtrng_scan+0x19/0x30 [virtio_rng]
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.505152]  virtio_dev_probe+0x1f9/0x2c0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.506096]  really_probe+0x1d1/0x3a0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.506958]  __driver_probe_device+0x11b/0x190
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.507993]  driver_probe_device+0x24/0xd0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.508955]  __driver_attach+0xd8/0x200
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.509859]  ? __device_attach_driver+0x120/0x120
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.510952]  bus_for_each_dev+0x7c/0xc0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.511846]  driver_attach+0x1e/0x30
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.512692]  bus_add_driver+0x178/0x220
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.513597]  driver_register+0x8f/0xf0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.514474]  ? 0xffffffffc0282000
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.515257]  register_virtio_driver+0x20/0x40
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.516277]  virtio_rng_driver_init+0x15/0x1000 [virtio_rng]
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.517602]  do_one_initcall+0x4a/0x210
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.518498]  ? kmem_cache_alloc_trace+0x181/0x300
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.519587]  ? do_init_module+0x27/0x200
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.520516]  do_init_module+0x50/0x200
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.521403]  load_module+0xb60/0xcb0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.522243]  __do_sys_finit_module+0xbd/0x130
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.523255]  ? __do_sys_finit_module+0xbd/0x130
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.524312]  __x64_sys_finit_module+0x18/0x20
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.525343]  do_syscall_64+0x5b/0x80
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.526183]  ? syscall_exit_to_user_mode+0x26/0x50
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.527287]  ? __x64_sys_mmap+0x33/0x50
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.528180]  ? do_syscall_64+0x67/0x80
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.529064]  ? exit_to_user_mode_prepare+0x30/0xa0
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.530182]  ? syscall_exit_to_user_mode+0x26/0x50
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.531289]  ? __x64_sys_newfstatat+0x1c/0x30
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.532310]  ? do_syscall_64+0x67/0x80
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.533185]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.534354] RIP: 0033:0x7f66accbf9fd
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.535191] RSP: 002b:00007ffc83fb0058 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.536912] RAX: ffffffffffffffda RBX: 0000555b12b044d0 RCX: 00007f66accbf9fd
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.538541] RDX: 0000000000000000 RSI: 00007f66ace41458 RDI: 0000000000000005
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.540161] RBP: 00007f66ace41458 R08: 0000000000000000 R09: 0000000000000000
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.541803] R10: 0000000000000005 R11: 0000000000000246 R12: 0000000000020000
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.543431] R13: 0000555b12b03e60 R14: 0000000000000000 R15: 0000555b12afff80
  Sep  9 22:31:32 bdmurray-test-systemd kernel: [ 1330.545063]  </TASK>

  This first occurred with at around 2022-08-24 with at least kernel
  version 5.19.0-15-generic. But  around 2022-09-07 started becoming
  much more frequent. I've downgraded the version of systemd to
  251.2-2ubuntu2 and that did not cause the oops to go away.

  ProblemType: Bug
  DistroRelease: Ubuntu 22.10
  Package: linux-image-5.19.0-15-generic 5.19.0-15.15
  ProcVersionSignature: Ubuntu 5.19.0-15.15-generic 5.19.0
  Uname: Linux 5.19.0-15-generic x86_64
  AlsaDevices:
   total 0
   crw-rw---- 1 root audio 116,  1 Sep  9 22:30 seq
   crw-rw---- 1 root audio 116, 33 Sep  9 22:30 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.23.0-0ubuntu1
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
  CRDA: N/A
  CasperMD5CheckResult: unknown
  Date: Sat Sep 10 15:51:33 2022
  Ec2AMI: ami-00009dfb
  Ec2AMIManifest: FIXME
  Ec2AvailabilityZone: nova
  Ec2InstanceType: autopkgtest
  Ec2Kernel: unavailable
  Ec2Ramdisk: unavailable
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  Lsusb: Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  Lsusb-t: /:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
  MachineType: OpenStack Foundation OpenStack Nova
  PciMultimedia:
   
  ProcEnviron:
   LC_CTYPE=en_US.UTF-8
   TERM=screen.xterm-256color
   PATH=(custom, no user)
   LANG=C.UTF-8
   SHELL=/bin/bash
  ProcFB:
   
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.19.0-15-generic root=UUID=63a35cf8-4d22-4a79-a93d-eb178abc8c07 ro console=ttyS0
  RelatedPackageVersions:
   linux-restricted-modules-5.19.0-15-generic N/A
   linux-backports-modules-5.19.0-15-generic  N/A
   linux-firmware                             20220831.gitd3c92280-0ubuntu1
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
  SourcePackage: linux
  UpgradeStatus: No upgrade log present (probably fresh install)
  dmi.bios.date: 04/01/2014
  dmi.bios.release: 0.0
  dmi.bios.vendor: SeaBIOS
  dmi.bios.version: Ubuntu-1.8.2-1ubuntu1+esm1
  dmi.chassis.type: 1
  dmi.chassis.vendor: QEMU
  dmi.chassis.version: pc-i440fx-xenial
  dmi.modalias: dmi:bvnSeaBIOS:bvrUbuntu-1.8.2-1ubuntu1+esm1:bd04/01/2014:br0.0:svnOpenStackFoundation:pnOpenStackNova:pvr13.1.4:cvnQEMU:ct1:cvrpc-i440fx-xenial:sku:
  dmi.product.family: Virtual Machine
  dmi.product.name: OpenStack Nova
  dmi.product.version: 13.1.4
  dmi.sys.vendor: OpenStack Foundation

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/1989258/+subscriptions