canonical-ubuntu-qa team mailing list archive

[Skia <2012667@xxxxxxxxxxxxxxxxxx>]

This was fixed in https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/274
The new code has been deployed for a while now. Closing as "Fixed released".

** Changed in: auto-package-testing
       Status: New => Fix Released

-- 
You received this bug notification because you are a member of
Canonical's Ubuntu QA, which is subscribed to Auto Package Testing.
https://bugs.launchpad.net/bugs/2012667

Title:
  autopkgtest-cloud running out of space is not handled well

Status in Auto Package Testing:
  Fix Released

Bug description:
  Some autopkgtests were running on autopkgtest-cloud-worker-lrg/7 when
  it ran out of disk space. From the test log we can see:

  autopkgtest [16:44:34]: testing package golang-github-bep-overlayfs version 0.6.0-2
  autopkgtest [16:44:34]: ERROR: unexpected error:
  OSError: [Errno 28] No space left on device

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 843, in main
      process_actions()
    File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 767, in process_actions
      tests_tree = build_source(kind, arg, built_binaries)
    File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 572, in build_source
      f.write('%s %s\n' % (testpkg_name, testpkg_version))
  OSError: [Errno 28] No space left on device

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 855, in <module>
      main()
    File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 845, in main
      errorcode = print_exception(sys.exc_info(), '')
    File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 248, in print_exception
      adtlog.psummary('quitting: unexpected error, see log')
    File "/home/ubuntu/autopkgtest/lib/adtlog.py", line 101, in psummary
      summary_stream.write(m.encode('UTF-8'))
  OSError: [Errno 28] No space left on device

  And this the journal from around that time frame:

  Mar 22 16:44:30 lrg-root4 /home/ubuntu/autopkgtest-cloud/worker/worker[706687]: INFO: autopkgtest exited with code 20
  Mar 22 16:44:30 lrg-root4 sh[706687]: OSError: [Errno 28] No space left on device
  Mar 22 16:44:30 lrg-root4 sh[706687]: During handling of the above exception, another exception occurred:
  Mar 22 16:44:30 lrg-root4 sh[706687]: Traceback (most recent call last):
  Mar 22 16:44:30 lrg-root4 sh[706687]:   File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 1123, in <module>
  Mar 22 16:44:30 lrg-root4 sh[706687]:     main()
  Mar 22 16:44:30 lrg-root4 sh[706687]:   File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 1116, in main
  Mar 22 16:44:30 lrg-root4 sh[706687]:     queue.wait()
  Mar 22 16:44:30 lrg-root4 sh[706687]:   File "/usr/lib/python3/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait
  Mar 22 16:44:30 lrg-root4 sh[706687]:     return self.dispatch_method(method_sig, args, content)
  Mar 22 16:44:30 lrg-root4 sh[706687]:   File "/usr/lib/python3/dist-packages/amqplib/client_0_8/abstract_channel.py", line 117, in dispatc
  h_method
  Mar 22 16:44:30 lrg-root4 sh[706687]:     return amqp_method(self, args, content)
  Mar 22 16:44:30 lrg-root4 sh[706687]:   File "/usr/lib/python3/dist-packages/amqplib/client_0_8/channel.py", line 2060, in _basic_deliver
  Mar 22 16:44:30 lrg-root4 sh[706687]:     func(msg)
  Mar 22 16:44:30 lrg-root4 sh[706687]:   File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 882, in request
  Mar 22 16:44:30 lrg-root4 sh[706687]:     f.write('%i\n' % code)
  Mar 22 16:44:30 lrg-root4 sh[706687]: OSError: [Errno 28] No space left on device
  Mar 22 16:44:30 lrg-root4 systemd[1]: autopkgtest@bos02-arm64-21.service: Main process exited, code=exited, status=1/FAILURE

  As I see it there are two problems here:
  1) the log files are left in /tmp/autopkgtest-work.$TEMP/out/
  2) the autopkgtest process itself can still be left running e.g.

  ubuntu    716051  0.0  0.0  45316 20736 ?        S    Mar22   0:00 /usr/bin/python3 -u /home/ubuntu/autopkgtest/runner/autopkgtest --outpu
  t-dir /tmp/autopkgtest-work.d5zzl576/out --timeout-copy=6000 -a i386 --setup-commands /home/ubuntu/autopkgtest-cloud/worker-config-product
  ion/setup-canonical.sh --setup-commands /home/ubuntu/autopkgtest/setup-commands/setup-testbed --apt-pocket=proposed=src:golang-golang-x-to
  ols --apt-upgrade golang-github-bep-overlayfs --timeout-short=300 --timeout-copy=20000 --timeout-build=20000 --env=ADT_TEST_TRIGGERS=golan
  g-golang-x-tools/1:0.5.0+ds-2 -- ssh -s /home/ubuntu/autopkgtest/ssh-setup/nova -- --flavor autopkgtest --security-groups autopkgtest-lrg-
  root4@lgw01-13.secgroup --name adt-lunar-i386-golang-github-bep-overlayfs-20230322-162349-lrg-root4 --image adt/ubuntu-lunar-amd64-server 
  --keyname testbed-lrg-root4 --net-id=net_prod-proposed-migration -e TERM=linux -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy=
  http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,ports.ubuntu.com,
  security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpad.net,10.24.0.0/24' --mirror=http://ftpmaster.internal/ubuntu/

To manage notifications about this bug go to:
https://bugs.launchpad.net/auto-package-testing/+bug/2012667/+subscriptions