yahoo-eng-team team mailing list archive

[James Falcon <2013419@xxxxxxxxxxxxxxxxxx>]

Tracked in Github Issues as https://github.com/canonical/cloud-
init/issues/4091

** Bug watch added: github.com/canonical/cloud-init/issues #4091
   https://github.com/canonical/cloud-init/issues/4091

** Changed in: cloud-init
       Status: Incomplete => Expired

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/2013419

Title:
  cc_grub_dpkg updates grub-pc or grub-efi debconf keys, but both can
  become incorrect on BIOS-booted Azure Ubuntu

Status in cloud-init:
  Expired

Bug description:
  Platform: Azure (generation 1 VMs)
  Image used: Ubuntu Server, SKU 20_04-lts (gen 1)
  (cloudinit 22.4.2-0ubuntu0~20.04.2 - no cloudinit customization)

  Azure generation 1 VMs boot in BIOS mode. Ubuntu comes with both BIOS and UEFI support installed, and cloudinit updates some debconf keys presumably to avoid mismatches when boot packages get updated on new machines:
  https://github.com/canonical/cloud-init/blob/ubuntu/22.4.2-0ubuntu0_20.04.2/cloudinit/config/cc_grub_dpkg.py#L148-L149

  Even when booted in BIOS mode, updating EFI packages (e.g. grub-efi-amd64-signed or shim-signed) will cause the debconf `grub-efi/install_devices` keys to be updated.
  If a discrepancy occurs on the disk ID where GRUB is installed (one scenario below), cloudinit only updates the `grub-pc` debconf keys (link above). The mismatched `grub-efi` key can cause further EFI package upgrades to fail, requiring a user with a shell to validate a prompt for dpkg configuration.


  [scenario]
  Sample scenario where we encountered this issue, using Packer to build a custom VM image:
  * Packer creates a BIOS VM from the base Ubuntu 20.04 image (gen 1).
  * cloudinit updates the `grub-pc` key:
  ```
  2023-02-26 08:40:19,507 - cc_grub_dpkg.py[DEBUG]: Setting grub debconf-set-selections with '/dev/disk/by-id/scsi-14d534654202020204f19e7ec574d624f9e27ff405f501bc0','false'
  ```
  * Packages get upgraded. Upgrades to EFI packages results in "Installing grub to /boot/efi" (dpkg logs) and debconf `grub-efi/install_devices` keys to be set, pointing at the Packer VM disk.
  * Customized VM gets saved by Packer as an image.

  ...

  * Later, we spin up gen 1 (BIOS) VMs from that image. Its root disk has its own serial ID.
    (GRUB partition = scsi-14d53465420202020da118904a05ed740b387a530ae506ac2-part15)
  * cloudinit updates the `grub-pc` key:
  ```
  2023-03-07 00:25:44,780 - cc_grub_dpkg.py[DEBUG]: Setting grub debconf-set-selections with '/dev/disk/by-id/scsi-14d534654202020200e6290db5a56ef43ba1f16eef596d653','false'
  ```
  * Later, a headless `apt upgrade` breaks:
  ```
  Setting up shim-signed (1.40.9+15.7-0ubuntu1) ...
  mount: /var/lib/grub/esp: special device /dev/disk/by-id/scsi-14d534654202020204f19e7ec574d624f9e27ff405f501bc0-part15 does not exist.
  ```
  ```
  # debconf-show grub-pc | egrep "grub-efi/install_devices:|grub-pc/install_devices:"
  * grub-efi/install_devices: /dev/disk/by-id/scsi-14d534654202020204f19e7ec574d624f9e27ff405f501bc0-part15
  * grub-pc/install_devices: /dev/disk/by-id/scsi-14d534654202020200e6290db5a56ef43ba1f16eef596d653
  ```
  [/scenario]


  In this situation, when running `apt upgrade` updating an EFI package (or `dpkg --configure -a` once broken) in a shell, a user can manually validate this prompt:
  ```
  ┌───────────────────────────────────┤ Configuring shim-signed ├────────────────────────────────────┐
  │ The GRUB boot loader was previously installed to a disk that is no longer present, or whose      |
  | unique identifier has changed for some reason. It is important to make sure that the installed   |
  | GRUB core image stays in sync with GRUB modules and grub.cfg. Please check again to make sure    |
  | that GRUB is written to the appropriate boot devices.                                            |
  │                                                                                                  |
  │ GRUB install devices:                                                                            |
  │                                                                                                  |
  │    [*] /dev/sda15 (111 MB; /boot/efi) on 32213 MB Virtual_Disk                                   |
  │                                                                                                  |
  │                                                                                                  │
  │                                              <Ok>                                                │
  │                                                                                                  │
  └──────────────────────────────────────────────────────────────────────────────────────────────────┘
  ```
  Accepting this prompt appears to update/fix the `grub-efi` debconf key.
  In my testing, `DEBIAN_FRONTEND=noninteractive` disables the prompt but it instead blows up with the aforementioned `mount: /var/lib/grub/esp...` error, which may be related to https://bugs.launchpad.net/ubuntu/+source/shim-signed/+bug/1940723 --- a bug suspiciously related to Azure VMs.

  Reminder: this is all on BIOS-booted VMs, as far as I know UEFI boot
  is never involved here.


  
  This bug is a follow-up to a quick discussion on https://github.com/canonical/cloud-init/commit/2fd24cc8cb2e2d1b0e00eb8c66573722523a91e7
  Support for EFI-booted machines to update grub debconf was introduced in that recent change, although based on the boot mode: if EFI-booted, update debconf `grub-efi`, otherwise update `grub-pc`.

  This unfortunately doesn't solve the case above, where BIOS machines
  have EFI configured and an intermediate/customized image is used.

  
  My uneducated guess is that we may want cloudinit to update either/both debconf keys if BIOS and/or EFI support is *installed*, instead of checking the current boot mode (= presence of `/sys/firmware/efi`).
  I do not know how to detect this. Presence of a grub-efi* package? Presence of /boot/efi?

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/2013419/+subscriptions