debcrafters-packages team mailing list archive

[Rocko <2116815@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Creating Ubuntu LXC containers with `lxc-ubuntu` template for some older
unsupported or ESM container releases fails with "Release signed by
unknown key". This is a regression in Oracular and newer.

## Steps to reproduce

$ sudo apt update
$ sudo apt install lxc lxc-templates
$ sudo lxc-create -n test-focal -t /usr/share/lxc/templates/lxc-ubuntu -- -r focal

## Expected outcome

LXC container creation succeeds. Partial output:

 Checking cache download in /var/cache/lxc/focal/rootfs-arm64 ...
 Installing packages in template: apt-transport-https,ssh,vim,language-pack-en
 Downloading ubuntu focal minimal ...
 I: Target architecture can be executed
 I: Retrieving InRelease
 I: Checking Release signature
 I: Valid Release signature (key id F6ECB3762474EDA9D21B7022871920D1991BC93C)
 I: Retrieving Packages
 I: Validating Packages
 I: Retrieving Packages
 I: Validating Packages
 I: Resolving dependencies of required packages...
 I: Resolving dependencies of base packages...
 I: Checking component main on http://ports.ubuntu.com/ubuntu-ports...
 I: Checking component universe on http://ports.ubuntu.com/ubuntu-ports...
 <snip>

Example output above is for a Noble host.

## Actual outcome

LXC container creation fails with "Release signed by unknown key":

 Checking cache download in /var/cache/lxc/focal/rootfs-arm64 ...
 Installing packages in template: apt-transport-https,ssh,vim,language-pack-en
 Downloading ubuntu focal minimal ...
 I: Target architecture can be executed
 I: Retrieving InRelease
 I: Checking Release signature
 E: Release signed by unknown key (key id 871920D1991BC93C)
    The specified keyring /usr/share/keyrings/ubuntu-archive-removed-keys.gpg may be incorrect or out of date.
    You can find the latest Debian release key at https://ftp-master.debian.org/keys.html
 lxc-create: test-focal: ../src/lxc/lxccontainer.c: create_run_template: 1601 Failed to create container from template
 lxc-create: test-focal: ../src/lxc/tools/lxc_create.c: lxc_create_main: 318 Failed to create container test-focal

Example output above is for a Questing host.

## Preliminary analysis

debootstrap 1.0.135, more specifically commit cc4962f03977
("Use correct keyring and mirror for Ubuntu"), changes the way debootstrap
determines what the default mirror address and keyring are for the specified
Ubuntu release. This commit switches to using the `--supported` list of versions
from `ubuntu-distro-data` command output for determining both default mirror and
keyring. It is not a 1-to-1 mapping however, and has multiple issues:

* Not all unsupported releases are moved to `old-releases.ubuntu.com`. This
  includes Trusty (end of ESM) and all active ESM releases. In such cases it
  requires explicitly specifying the mirror URL when calling debootstrap.
* The location of the GPG key used to sign the release is independent of its
  support status. In such cases it would require to manually specify the keyring
  path when calling debootstrap.

The mismatched keyring situation is explicitly mentioned in the comments in
`scripts/gutsy` in the `debootstrap` package:

 # Note: The "removed keys" does not automatically tally with unsupported releases.
 # In this case the user will need to use the --keyring= switch.

Nothing is said about the default mirror, which suffers from the same problem.
There is no guarantee that unsupported releases are immediately moved
from `archive.ubuntu.com` to `old-releases.ubuntu.com`, Trusty is a good example
of that.

The `lxc-ubuntu` template does not support passing a custom keyring path to
`debootstrap`.

Additionally, I checked the `Release` files for all known releases on
`archive.ubuntu.com` and `old-releases.ubuntu.com` to figure out where each
release resides and which signing key(s) it uses. The summarised spreadsheet is
attached in `debootstrap_releases.png`.

## Proposed fixes

I see a few possible points of improvement here:

1. In `debootstrap`, check `--supported-esm` in addition to `--supported` to
   determine the default mirror address.
2. Define the minimal release that uses a key from `ubuntu-archive-keyring.gpg`,
   and use that as a guide to determine which keyring to use when one is not
   specified. Not sure about where to store this information however.
3. Move Trusty from `archive.ubuntu.com` to `old-releases.ubuntu.com` for the
   sake of consistency. It has reached end of ESM over a year ago (2024-04-25).
4. Add a `--keyring` option to the `lxc-ubuntu` template to support a custom
   keyring path that is then passed to `debootstrap`.

Looking for feedback on these, as well as ideas on how to approach this in a
better way. Thanks!

## Environment

Tested on Noble and Questing arm64 virtual machines via Parallels
Desktop on macOS 15.5.

Working configuration: Noble
lxc				1:5.0.3-2ubuntu7.2
lxc-templates	3.0.4.79.g84b0597-1
distro-info		1.7build1
debootstrap		1.0.134ubuntu1
ubuntu-keyring	2023.11.28.1

Broken configuration: Questing
lxc				1:6.0.4-4ubuntu1
lxc-templates	3.0.4.89.gc128b96-1
distro-info		1.13
debootstrap		1.0.141
ubuntu-keyring	2023.11.28.1

** Affects: debootstrap (Ubuntu)
     Importance: Undecided
         Status: New

** Affects: lxc-templates (Ubuntu)
     Importance: Undecided
         Status: New

** Attachment added: "Summary of available Ubuntu releases, their default mirror and signing keys"
   https://bugs.launchpad.net/bugs/2116815/+attachment/5889542/+files/debootstrap_releases.png

** Also affects: debootstrap (Ubuntu)
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of
Debcrafters packages, which is subscribed to debootstrap in Ubuntu.
https://bugs.launchpad.net/bugs/2116815

Title:
  Container creation fails on "lxc-ubuntu" template for some
  unsupported/ESM container releases

Status in debootstrap package in Ubuntu:
  New
Status in lxc-templates package in Ubuntu:
  New

Bug description:
  Creating Ubuntu LXC containers with `lxc-ubuntu` template for some older
  unsupported or ESM container releases fails with "Release signed by
  unknown key". This is a regression in Oracular and newer.

  ## Steps to reproduce

  $ sudo apt update
  $ sudo apt install lxc lxc-templates
  $ sudo lxc-create -n test-focal -t /usr/share/lxc/templates/lxc-ubuntu -- -r focal

  ## Expected outcome

  LXC container creation succeeds. Partial output:

   Checking cache download in /var/cache/lxc/focal/rootfs-arm64 ...
   Installing packages in template: apt-transport-https,ssh,vim,language-pack-en
   Downloading ubuntu focal minimal ...
   I: Target architecture can be executed
   I: Retrieving InRelease
   I: Checking Release signature
   I: Valid Release signature (key id F6ECB3762474EDA9D21B7022871920D1991BC93C)
   I: Retrieving Packages
   I: Validating Packages
   I: Retrieving Packages
   I: Validating Packages
   I: Resolving dependencies of required packages...
   I: Resolving dependencies of base packages...
   I: Checking component main on http://ports.ubuntu.com/ubuntu-ports...
   I: Checking component universe on http://ports.ubuntu.com/ubuntu-ports...
   <snip>

  Example output above is for a Noble host.

  ## Actual outcome

  LXC container creation fails with "Release signed by unknown key":

   Checking cache download in /var/cache/lxc/focal/rootfs-arm64 ...
   Installing packages in template: apt-transport-https,ssh,vim,language-pack-en
   Downloading ubuntu focal minimal ...
   I: Target architecture can be executed
   I: Retrieving InRelease
   I: Checking Release signature
   E: Release signed by unknown key (key id 871920D1991BC93C)
      The specified keyring /usr/share/keyrings/ubuntu-archive-removed-keys.gpg may be incorrect or out of date.
      You can find the latest Debian release key at https://ftp-master.debian.org/keys.html
   lxc-create: test-focal: ../src/lxc/lxccontainer.c: create_run_template: 1601 Failed to create container from template
   lxc-create: test-focal: ../src/lxc/tools/lxc_create.c: lxc_create_main: 318 Failed to create container test-focal

  Example output above is for a Questing host.

  ## Preliminary analysis

  debootstrap 1.0.135, more specifically commit cc4962f03977
  ("Use correct keyring and mirror for Ubuntu"), changes the way debootstrap
  determines what the default mirror address and keyring are for the specified
  Ubuntu release. This commit switches to using the `--supported` list of versions
  from `ubuntu-distro-data` command output for determining both default mirror and
  keyring. It is not a 1-to-1 mapping however, and has multiple issues:

  * Not all unsupported releases are moved to `old-releases.ubuntu.com`. This
    includes Trusty (end of ESM) and all active ESM releases. In such cases it
    requires explicitly specifying the mirror URL when calling debootstrap.
  * The location of the GPG key used to sign the release is independent of its
    support status. In such cases it would require to manually specify the keyring
    path when calling debootstrap.

  The mismatched keyring situation is explicitly mentioned in the comments in
  `scripts/gutsy` in the `debootstrap` package:

   # Note: The "removed keys" does not automatically tally with unsupported releases.
   # In this case the user will need to use the --keyring= switch.

  Nothing is said about the default mirror, which suffers from the same problem.
  There is no guarantee that unsupported releases are immediately moved
  from `archive.ubuntu.com` to `old-releases.ubuntu.com`, Trusty is a good example
  of that.

  The `lxc-ubuntu` template does not support passing a custom keyring path to
  `debootstrap`.

  Additionally, I checked the `Release` files for all known releases on
  `archive.ubuntu.com` and `old-releases.ubuntu.com` to figure out where each
  release resides and which signing key(s) it uses. The summarised spreadsheet is
  attached in `debootstrap_releases.png`.

  ## Proposed fixes

  I see a few possible points of improvement here:

  1. In `debootstrap`, check `--supported-esm` in addition to `--supported` to
     determine the default mirror address.
  2. Define the minimal release that uses a key from `ubuntu-archive-keyring.gpg`,
     and use that as a guide to determine which keyring to use when one is not
     specified. Not sure about where to store this information however.
  3. Move Trusty from `archive.ubuntu.com` to `old-releases.ubuntu.com` for the
     sake of consistency. It has reached end of ESM over a year ago (2024-04-25).
  4. Add a `--keyring` option to the `lxc-ubuntu` template to support a custom
     keyring path that is then passed to `debootstrap`.

  Looking for feedback on these, as well as ideas on how to approach this in a
  better way. Thanks!

  ## Environment

  Tested on Noble and Questing arm64 virtual machines via Parallels
  Desktop on macOS 15.5.

  Working configuration: Noble
  lxc				1:5.0.3-2ubuntu7.2
  lxc-templates	3.0.4.79.g84b0597-1
  distro-info		1.7build1
  debootstrap		1.0.134ubuntu1
  ubuntu-keyring	2023.11.28.1

  Broken configuration: Questing
  lxc				1:6.0.4-4ubuntu1
  lxc-templates	3.0.4.89.gc128b96-1
  distro-info		1.13
  debootstrap		1.0.141
  ubuntu-keyring	2023.11.28.1

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/debootstrap/+bug/2116815/+subscriptions