canonical-ubuntu-qa team mailing list archive

[Nick Rosbrook <mp+459538@xxxxxxxxxxxxxxxxxx>]

Nick Rosbrook has proposed merging ~enr0n/autopkgtest:deb822 into ~ubuntu-release/autopkgtest/+git/development:master.

Requested reviews:
  Ubuntu Release Team (ubuntu-release)

For more details, see:
https://code.launchpad.net/~enr0n/autopkgtest/+git/autopkgtest/+merge/459538

Add some initial support for deb822 sources, specifically in the setup-testbed script. These patches will also be needed in the noble deb because users running autopkgtest locally on noble could hit this too. For users running locally, I think the autopkgtest-build* scripts might be need updating too, but I am not sure that's relevant for production.

I first saw this in systemd's test-in-lxd test[1], which runs autopkgtest manually from within a container, where the following error is seen:

4072s Running setup script /usr/share/autopkgtest/setup-commands/setup-testbed...
4072s awk: fatal: cannot open file `/etc/apt/sources.list' for reading: No such file or directory
4072s sh: Attempting to set up Debian/Ubuntu apt sources automatically
4072s Failed to auto-detect apt mirror; set $MIRROR explicitly
4075s patching file /usr/bin/autopkgtest-build-lxd
4075s Creating autopkgtest-prepare-PkU
4084s 

This happens because the "host" in this case is a noble container with /etc/apt/sources.list.d/ubuntu.sources. The only test I have done is running tests-in-lxd locally using a PPA version of autopkgtest[2].

[1] https://autopkgtest.ubuntu.com/results/autopkgtest-noble/noble/amd64/s/systemd/20240126_041110_52227@/log.gz
[2] https://launchpad.net/~enr0n/+archive/ubuntu/systemd/+packages?field.name_filter=autopkgtest&field.status_filter=published&field.series_filter=
-- 
Your team Canonical's Ubuntu QA is subscribed to branch ~ubuntu-release/autopkgtest/+git/development:master.

diff --git a/setup-commands/setup-testbed b/setup-commands/setup-testbed
index 553b30e..75c8c31 100755
--- a/setup-commands/setup-testbed
+++ b/setup-commands/setup-testbed
@@ -184,7 +184,20 @@ if [ -e "$root/etc/os-release" ]; then
     DISTRO_ID=$(. "$root/etc/os-release" && echo "$ID" || echo INVALID)
 fi
 if [ -z "${MIRROR:-}" ]; then
-    MIRROR=$(awk '/^deb .*'"$DISTRO_ID"'/ { sub(/\[.*\]/, "", $0); print $2; exit }' "$root/etc/apt/sources.list")
+    if [ -f "/etc/apt/sources.list" ]; then
+        # shellcheck disable=SC2016
+        MIRROR=$(chroot "$root" awk '/^deb .*'"$DISTRO_ID"'/ { sub(/\[.*\]/, "", $0); print $2; exit }' "/etc/apt/sources.list" || :)
+        deb822=
+    elif [ -f "/etc/apt/sources.list.d/$DISTRO_ID.sources" ]; then
+        # Starting with 24.04, Ubuntu uses /etc/apt/sources.list.d/ubuntu.sources
+        # for default sources, so check for that here.
+        # shellcheck disable=SC2016
+        MIRROR=$(chroot "$root" awk '/^URIs: .*'"$DISTRO_ID"'/ { sub(/\[.*\]/, "", $0); print $2; exit }' "/etc/apt/sources.list.d/$DISTRO_ID.sources" || :)
+        deb822="y"
+    fi
+fi
+if [ -z "${RELEASE:-}" ]; then
+    RELEASE=$(. "$root/etc/os-release" && echo "$VERSION_CODENAME" || echo "")
 fi
 if [ -z "${RELEASE:-}" ]; then
     RELEASE=$(awk '/^deb .*'"$DISTRO_ID"'/ { sub(/\[.*\]/, "", $0); print $3; exit }' "$root/etc/apt/sources.list")
@@ -194,10 +207,22 @@ if [ -n "${AUTOPKGTEST_KEEP_APT_SOURCES:-}" ]; then
     echo "$0: Keeping existing apt sources" >&2
 elif [ -n "${AUTOPKGTEST_APT_SOURCES_FILE:-}" ]; then
     echo "$0: Copying apt sources from $AUTOPKGTEST_APT_SOURCES_FILE" >&2
-    install -m644 "$AUTOPKGTEST_APT_SOURCES_FILE" "$root/etc/apt/sources.list"
+    install -m644 "$AUTOPKGTEST_APT_SOURCES_FILE" "$root/etc/apt/sources.list.d/"
 elif [ -n "${AUTOPKGTEST_APT_SOURCES:-}" ]; then
     echo "$0: Setting apt sources from \$AUTOPKGTEST_APT_SOURCES" >&2
-    printf '%s\n' "$AUTOPKGTEST_APT_SOURCES" > "$root/etc/apt/sources.list"
+
+    if echo "$AUTOPKGTEST_APT_SOURCES" | grep -Eq "^deb|^deb-src"; then
+        # These look like .list format.
+        printf '%s\n' "$AUTOPKGTEST_APT_SOURCES" > "$root/etc/apt/sources.list.d/autopkgtest.list"
+    elif echo "$AUTOPKGTEST_APT_SOURCES" | grep -q "^Types:"; then
+        # These look like .sources format.
+        printf '%s\n' "$AUTOPKGTEST_APT_SOURCES" > "$root/etc/apt/sources.list.d/autopkgtest.sources"
+    else
+        # Deliberately not expanding $AUTOPKGTEST_APT_SOURCES here
+        # shellcheck disable=SC2016
+        echo 'Failed to determine apt sources format used in $AUTOPKGTEST_APT_SOURCES.' >&2
+        exit 1
+    fi
 else
     echo "$0: Attempting to set up Debian/Ubuntu apt sources automatically" >&2
 
@@ -215,7 +240,17 @@ else
     fi
     if [ "${MIRROR%ubuntu*}" != "$MIRROR" ]; then
         echo "$0: Distribution appears to be Ubuntu" >&2
-        cat << EOF > "$root/etc/apt/sources.list"
+
+        if [ -n "$deb822" ]; then
+            cat << EOF > "$root/etc/apt/sources.list.d/$DISTRO_ID.sources"
+
+Types: deb deb-src
+URIs: $MIRROR
+Suites: ${RELEASE} ${RELEASE}-updates ${RELEASE}-security
+Components: main restricted universe multiverse
+EOF
+        else
+            cat << EOF > "$root/etc/apt/sources.list"
 deb     $MIRROR ${RELEASE} main restricted universe multiverse
 deb     $MIRROR ${RELEASE}-updates main restricted universe multiverse
 deb     $MIRROR ${RELEASE}-security main restricted universe multiverse
@@ -223,12 +258,24 @@ deb-src $MIRROR ${RELEASE} main restricted universe multiverse
 deb-src $MIRROR ${RELEASE}-updates main restricted universe multiverse
 deb-src $MIRROR ${RELEASE}-security main restricted universe multiverse
 EOF
+        fi
     else
         echo "$0: Distribution assumed to resemble Debian" >&2
-        cat << EOF > "$root/etc/apt/sources.list"
+
+        if [ -n "$deb822" ]; then
+            cat << EOF > "$root/etc/apt/sources.list.d/$DISTRO_ID.sources"
+
+Types: deb deb-src
+URIs: $MIRROR
+Suites: $RELEASE
+Components: main contrib non-free
+EOF
+        else
+            cat << EOF > "$root/etc/apt/sources.list"
 deb     $MIRROR $RELEASE main contrib non-free
 deb-src $MIRROR $RELEASE main contrib non-free
 EOF
+        fi
     fi
 fi