canonical-ubuntu-qa team mailing list archive

[kip <2056461@xxxxxxxxxxxxxxxxxx>]

Hello Dominique,

I prepared an autopkgtest container like so with the 6.11-rc4 kernel
installed:

$ autopkgtest-buildvm-ubuntu-cloud --verbose -r noble -a amd64 --cloud-
image-url http://cloud-images.ubuntu.com/daily/server --post-command
"wget -c https://kernel.ubuntu.com/mainline/v6.11-rc4/amd64/linux-
headers-6.11.0-061100rc4-generic_6.11.0-061100rc4.202408182119_amd64.deb
https://kernel.ubuntu.com/mainline/v6.11-rc4/amd64/linux-
headers-6.11.0-061100rc4_6.11.0-061100rc4.202408182119_all.deb
https://kernel.ubuntu.com/mainline/v6.11-rc4/amd64/linux-image-
unsigned-6.11.0-061100rc4-generic_6.11.0-061100rc4.202408182119_amd64.deb
https://kernel.ubuntu.com/mainline/v6.11-rc4/amd64/linux-
modules-6.11.0-061100rc4-generic_6.11.0-061100rc4.202408182119_amd64.deb
&& sudo dpkg -i *.deb"

That builds a container image and then downloads the mainline kernel
with, unless I'm mistaken, the patch in there from:

https://kernel.ubuntu.com/mainline/v6.11-rc4/

I then attempt to perform CI testing for the GNU hello package with the
fresh test bed:

$ sudo TMPDIR=$PWD autopkgtest --apt-upgrade -ddd -U
./hello_2.10-3build1.dsc -- qemu --qemu-options=-enable-kvm --cpus=8
--ram-size=8192 --debug --show-boot autopkgtest-noble-amd64.img |& ts
%.T | /usr/bin/tee ./aptest.log

Like with my comments in #38, I noticed that networking was initially
dead - at least the first time I ran it. I was not running any special
setup commands to install or remove any packages in the test bed, other
than whatever the hello package is asking the package manager to pull.

I would have attached my aptest.log again but for good measure I wanted
to replicate the same issue. So I ran the command again, and again, and
again, and it seems to work now clobbering the old log. Let's hope it
stays that way and I'm not simply catching a race condition when it's in
a good mood.

-- 
You received this bug notification because you are a member of
Canonical's Ubuntu QA, which is subscribed to autopkgtest in Ubuntu.
https://bugs.launchpad.net/bugs/2056461

Title:
  autopkgtest-virt-qemu on noble images sometimes hangs doing copydown

Status in Linux:
  Confirmed
Status in autopkgtest package in Ubuntu:
  Fix Committed
Status in linux package in Ubuntu:
  In Progress
Status in autopkgtest package in Debian:
  Fix Released

Bug description:
  [Impact]

  It seems that kernel 6.8 introduced a regression in the 9pfs related
  to caching and netfslib, that can cause some user-space apps to read
  content from files that is not up-to-date (when they are used in a
  producer/consumer fashion).

  It seems that the offending commit is this one:

   80105ed2fd27 ("9p: Use netfslib read/write_iter")

  Reverting the commit seems to fix the problem. However the actual bug
  might be in netfslib or how netfslib is used in the 9p context.

  The regression has been reported upstream and we are still
  investigating (https://lore.kernel.org/lkml/Zj0ErxVBE3DYT2Ea@gpd/).

  In the meantime it probably makes sense to temporarily revert the
  commit as a SAUCE patch. Then we will drop the SAUCE patch once we'll
  have a proper fix upstream.

  [Test case]

  The following test should complete correctly without any timeout:

    pull-lp-source -d hello
    autopkgtest-buildvm-ubuntu-cloud -r noble
    autopkgtest -U hello*.dsc -- qemu ./autopkgtest-noble-amd64.img

  
  [Fix]

  Revert the following commit (until we have a proper fix upstream):

   80105ed2fd27 ("9p: Use netfslib read/write_iter")

  [Regression potential]

  We may experience other regressions related to 9pfs with this change,
  however it's quite unlikely to happen since we are reverting a commit,
  restoring the previous behavior.

  [Original bug report]

  autopkgtest-virt-qemu sometimes hangs when running tests on noble
  images. Originally reported by schopin, who also provided a
  reproducer:

    pull-lp-source -d hello
    autopkgtest-buildvm-ubuntu-cloud -r noble
    autopkgtest -U hello*.dsc -- qemu ./autopkgtest-noble-amd64.img

  I've been able to reproduce it with debugging enabled:

    autopkgtest -ddd -U hello_2.10-3.dsc -- qemu --debug --show-boot
  /path/to/image

  It can get stuck during different stages, but AFAICT always during
  "copydown" operations, log excerpts follow. It may be a coincidence,
  but this started happening around the time linux-
  image-6.8.0-11-generic (6.8.0-11.11) migrated to noble. The testbeds I
  used booted 6.6 but then rebooted into that 6.8 kernel after being
  upgraded by autopkgtest.

  -- logs --

  Removing autopkgtest-satdep (0) ...
  [...]
  autopkgtest-virt-qemu: DBG: executing copydown /tmp/autopkgtest.output.g8v75e8g/tests-tree/ /t/
  autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory', '/tmp/autopkgtest.output.g8v75e]
  autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader name='/dev/null'>", 'deststdout']
  autopkgtest-virt-qemu: DBG:  +< tar --directory /tmp/autopkgtest.output.g8v75e8g/tests-tree/ --
  autopkgtest-virt-qemu: DBG:  +> /tmp/autopkgtest-qemu.ztmr6f5k/runcmd sh -ec if ! test -d /tmp-
  autopkgtest-virt-qemu: DBG:  +>?

  -- or --

  autopkgtest: DBG: sending command to testbed: copydown /tmp/autopkgtest.output.c9utq3bx/tests-tree/ /tmp/autopkgtest.H8NDfW/build.DLR/src/
  autopkgtest-virt-qemu: DBG: executing copydown /tmp/autopkgtest.output.c9utq3bx/tests-tree/ /tmp/autopkgtest.H8NDfW/build.DLR/src/
  autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory', '/tmp/autopkgtest.output.c9utq3bx/tests-tree/', '--warning=none', '-c', '.', '-f', '-'], ['/tmp/autopkgtest-qemu.qtkcgg5l/runcm]
  autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader name='/dev/null'>", 'deststdout', "<_io.BufferedReader name='/dev/null'>", 'devnull_read', <_io.BufferedReader name='/dev/null'>]
  autopkgtest-virt-qemu: DBG:  +< tar --directory /tmp/autopkgtest.output.c9utq3bx/tests-tree/ --warning=none -c . -f -
  autopkgtest-virt-qemu: DBG:  +> /tmp/autopkgtest-qemu.qtkcgg5l/runcmd sh -ec if ! test -d /tmp/autopkgtest.H8NDfW/build.DLR/src/; then mkdir -- /tmp/autopkgtest.H8NDfW/build.DLR/src/; fi; cd-
  autopkgtest-virt-qemu: DBG:  +>?

To manage notifications about this bug go to:
https://bugs.launchpad.net/linux/+bug/2056461/+subscriptions