canonical-ubuntu-qa team mailing list archive

Thread
Date

[Bug 1988527] Re: autopkgtest_qemu doesn't use accel=kvm on ppc64le, being fully unusable on that arch

To: canonical-ubuntu-qa@xxxxxxxxxxxxxxxxxxx
From: Paride Legovini <1988527@xxxxxxxxxxxxxxxxxx>
Date: Fri, 16 Sep 2022 16:20:15 -0000
Reply-to: Bug 1988527 <1988527@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx

Note for the SRU team: there is some noise in the Jammy SRU debdiff [1]
caused by the fact that by default `dpkg-source -b` excludes .gitignore
from the tarball, but the Debian upload has been done using dgit, which
doesn't have any special exclude rules.

[1]
https://launchpadlibrarian.net/623539153/autopkgtest_5.20_5.20ubuntu1.diff.gz

** Description changed:

[ Impact ]

On Power9 the qemu based autopkgtest commands create VMs that are
extremely slow and fail with obscure errors (partially discussed in LP:
#1973628, comment 8). This can be reproduced for example by running:

autopkgtest-buildvm-ubuntu-cloud -v -r jammy --ram-size 1024

but autopkgtest-virt-qemu is also affected.

This happens because autopkgtest fails to detect the system architecture
as KVM capable due to a typo in the architecture name (ppc64el instead
of ppc64le). This upload fixes the typo.

Fixing this bug in Jammy will allow users and developers to manually run
autopkgtests on ppc64el. This is useful for example in +1 maintenance.

[ Test Plan ]

Run:

autopkgtest-buildvm-ubuntu-cloud -v -r jammy --ram-size 1024

- on an affected system.
+ on an affected system (= a KVM-capable POWER machine running Jammy).

Buggy package => the command takes hours to complete and prints lots of
obscure errors.

Fixed package => the command completes in minutes.

[ Where problems could occur ]

Without this fix qemu based autopkgtest could in principle complete even
when KVM is available (/dev/kvm exists) but broken, as it may be in some
nested virtualization scenarios. This said, without KVM qemu based
appears to be very broken due to timeouts caused by its extreme
slowness, so I think the risk of causing a regression is marginal.

[ Other Info ]

The very same fix has been submitted and merged upstream, and released
to Kinetic via a clean cherry-pick.

[ Original Description ]

autopkgtest-buildvm-ubuntu-cloud -v -r jammy --ram-size 1024

but autopkgtest-virt-qemu is also affected. The extreme slowness of the
VMs made me think that something was off with the virtualization
settings. I modified autopkgtest_qemu.py so that qemu-system-ppc64le is
called with '-machine accel=kvm' (which I think is the same as '-machine
pseries,accel=kvm' with pseries being the default machine type).

With this change everything is very fast and reliable. These warnings
also went away:

qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-cfpc=workaround
qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-sbbc=workaround
qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-ibs=workaround
qemu-system-ppc64le: warning: TCG doesn't support requested feature, cap-ccf-assist=on

indicating that we were using TCG emulation before.

I imagine that Qemu has good reasons not to default to accel=kvm or
accel=kvm:tcg on ppc64, but think it's reasonable to assume it's
available and enable it in autopkgtest.

We can fix this in autopkgtest upstream, but it would be nice to verify
if this is an issue with Debian too before submitting a salsa MR.

[1] https://wiki.qemu.org/Documentation/TCG

--
You received this bug notification because you are a member of
Canonical's Ubuntu QA, which is subscribed to the bug report.
https://bugs.launchpad.net/bugs/1988527

Title:
autopkgtest_qemu doesn't use accel=kvm on ppc64le, being fully
unusable on that arch

Status in autopkgtest package in Ubuntu:
Fix Released
Status in autopkgtest source package in Jammy:
In Progress
Status in autopkgtest package in Debian:
Fix Committed

Bug description:
[ Impact ]

On Power9 the qemu based autopkgtest commands create VMs that are
extremely slow and fail with obscure errors (partially discussed in
LP: #1973628, comment 8). This can be reproduced for example by
running:

autopkgtest-buildvm-ubuntu-cloud -v -r jammy --ram-size 1024

but autopkgtest-virt-qemu is also affected.

This happens because autopkgtest fails to detect the system
architecture as KVM capable due to a typo in the architecture name
(ppc64el instead of ppc64le). This upload fixes the typo.

Fixing this bug in Jammy will allow users and developers to manually
run autopkgtests on ppc64el. This is useful for example in +1
maintenance.

[ Test Plan ]

Run:

autopkgtest-buildvm-ubuntu-cloud -v -r jammy --ram-size 1024

on an affected system (= a KVM-capable POWER machine running Jammy).

Buggy package => the command takes hours to complete and prints lots
of obscure errors.

Fixed package => the command completes in minutes.

[ Where problems could occur ]

Without this fix qemu based autopkgtest could in principle complete
even when KVM is available (/dev/kvm exists) but broken, as it may be
in some nested virtualization scenarios. This said, without KVM qemu
based appears to be very broken due to timeouts caused by its extreme
slowness, so I think the risk of causing a regression is marginal.

[ Other Info ]

The very same fix has been submitted and merged upstream, and released
to Kinetic via a clean cherry-pick.

[ Original Description ]

On Power9 the qemu based autopkgtest commands create VMs that are
extremely slow and fail with obscure errors (partially discussed in
LP: #1973628, comment 8). This can be reproduced for example by
running:

autopkgtest-buildvm-ubuntu-cloud -v -r jammy --ram-size 1024

but autopkgtest-virt-qemu is also affected. The extreme slowness of
the VMs made me think that something was off with the virtualization
settings. I modified autopkgtest_qemu.py so that qemu-system-ppc64le
is called with '-machine accel=kvm' (which I think is the same as
'-machine pseries,accel=kvm' with pseries being the default machine
type).

With this change everything is very fast and reliable. These warnings
also went away:

indicating that we were using TCG emulation before.

I imagine that Qemu has good reasons not to default to accel=kvm or
accel=kvm:tcg on ppc64, but think it's reasonable to assume it's
available and enable it in autopkgtest.

We can fix this in autopkgtest upstream, but it would be nice to
verify if this is an issue with Debian too before submitting a salsa
MR.

[1] https://wiki.qemu.org/Documentation/TCG

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/autopkgtest/+bug/1988527/+subscriptions