canonical-ubuntu-qa team mailing list archive

[Skia <2056461@xxxxxxxxxxxxxxxxxx>]

Okay, I've spend the last 24h digging into that issue, and here are my findings:
* I confirm I can only reproduce the issue with 6.8 kernels, not with 6.5. Just to be sure, I've also tested 6.8.0-32.32 in kernel team's PPA, and it has the issue too.
* Changing `--cpus` or `--ram-size` in `autopkgtest-virt-qemu` doesn't change anything.
* The code responsible for this in autopkgtest is a wonderful piece of black magic: most lives in `virt/autopkgtest-virt-qemu`, and the main part is the `runcmd` script being crafted in the `make_auxverb` function. The hanging is the `# wait until command has exited` loop, that never exits.

The most funny part is the new workaround that I found.
Run the following from your host that runs `autopkgtest`. Obvious change the port number depending on which free port was available for `autopkgtest`.
```
ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -p 10022 ubuntu@localhost stat /run/autopkgtest/shared/job\*/stdin
```
For some reason, `stat`ing that file in the testbed makes the loop exit and everything continues as normal, meaning you can use the test results normally.
I've seen this work at least 20 times on my machine, but I'd like someone else to confirm that this workaround indeed works.

This bug is becoming more and more interesting, stay tuned for more
adventures!

-- 
You received this bug notification because you are a member of
Canonical's Ubuntu QA, which is subscribed to autopkgtest in Ubuntu.
https://bugs.launchpad.net/bugs/2056461

Title:
  autopkgtest-virt-qemu on noble images sometimes hangs doing copydown

Status in autopkgtest package in Ubuntu:
  Confirmed

Bug description:
  autopkgtest-virt-qemu sometimes hangs when running tests on noble
  images. Originally reported by schopin, who also provided a
  reproducer:

    pull-lp-source -d hello
    autopkgtest-buildvm-ubuntu-cloud -r noble
    autopkgtest -U hello*.dsc -- qemu ./autopkgtest-noble-amd64.img

  I've been able to reproduce it with debugging enabled:

    autopkgtest -ddd -U hello_2.10-3.dsc -- qemu --debug --show-boot
  /path/to/image

  It can get stuck during different stages, but AFAICT always during
  "copydown" operations, log excerpts follow. It may be a coincidence,
  but this started happening around the time linux-
  image-6.8.0-11-generic (6.8.0-11.11) migrated to noble. The testbeds I
  used booted 6.6 but then rebooted into that 6.8 kernel after being
  upgraded by autopkgtest.

  -- logs --

  Removing autopkgtest-satdep (0) ...
  [...]
  autopkgtest-virt-qemu: DBG: executing copydown /tmp/autopkgtest.output.g8v75e8g/tests-tree/ /t/
  autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory', '/tmp/autopkgtest.output.g8v75e]
  autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader name='/dev/null'>", 'deststdout']
  autopkgtest-virt-qemu: DBG:  +< tar --directory /tmp/autopkgtest.output.g8v75e8g/tests-tree/ --
  autopkgtest-virt-qemu: DBG:  +> /tmp/autopkgtest-qemu.ztmr6f5k/runcmd sh -ec if ! test -d /tmp-
  autopkgtest-virt-qemu: DBG:  +>?

  -- or --

  autopkgtest: DBG: sending command to testbed: copydown /tmp/autopkgtest.output.c9utq3bx/tests-tree/ /tmp/autopkgtest.H8NDfW/build.DLR/src/
  autopkgtest-virt-qemu: DBG: executing copydown /tmp/autopkgtest.output.c9utq3bx/tests-tree/ /tmp/autopkgtest.H8NDfW/build.DLR/src/
  autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory', '/tmp/autopkgtest.output.c9utq3bx/tests-tree/', '--warning=none', '-c', '.', '-f', '-'], ['/tmp/autopkgtest-qemu.qtkcgg5l/runcm]
  autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader name='/dev/null'>", 'deststdout', "<_io.BufferedReader name='/dev/null'>", 'devnull_read', <_io.BufferedReader name='/dev/null'>]
  autopkgtest-virt-qemu: DBG:  +< tar --directory /tmp/autopkgtest.output.c9utq3bx/tests-tree/ --warning=none -c . -f -
  autopkgtest-virt-qemu: DBG:  +> /tmp/autopkgtest-qemu.qtkcgg5l/runcmd sh -ec if ! test -d /tmp/autopkgtest.H8NDfW/build.DLR/src/; then mkdir -- /tmp/autopkgtest.H8NDfW/build.DLR/src/; fi; cd-
  autopkgtest-virt-qemu: DBG:  +>?

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/2056461/+subscriptions