← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #95010

[Bug 2091114] [NEW] Nova validation checks checks reject valid UEFI image

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This relates specifically to this image:
https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/flatcar/flatcar-stable-4081.2.0-kube-v1.30.1.img

However, the problem should be easy enough to understand just from the
description here without downloading it.

When attempting to boot the image in 2024.2 devstack we see the
following failure:

Dec 04 11:01:53 capo-e2e-controller.c.k8s-infra-e2e-boskos-107.internal
nova-compute[114399]: ESC[01;33mWARNING
oslo_utils.imageutils.format_inspector [ESC[01;36mNone
req-993c42cb-8da1-4cc5-83fd-1c16c08cbc13 ESC[00;36mdemo demoESC[01;33m]
ESC[01;35mESC[01;33mSafety check mbr on gpt failed because GPT MBR
defines invalid extra partitionsESC[00m:
oslo_utils.imageutils.format_inspector.SafetyViolation: GPT MBR defines
invalid extra partitionsESC[00m

There is an associated stack trace and the server enters the ERROR
state.

This is a QCOW2 image. After downloading it I can manually convert it to
raw to inspect its partition table:

> qemu-img convert -f qcow2 flatcar-stable-4081.2.0-kube-v1.30.1.img -O raw flatcar-stable-4081.2.0-kube-v1.30.1.img.raw
> fdisk -l flatcar-stable-4081.2.0-kube-v1.30.1.img.raw
Disk flatcar-stable-4081.2.0-kube-v1.30.1.img.raw: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: D814FAF6-AD0A-4FC1-8DE9-236755D902E5

Device                                          Start      End  Sectors  Size Type
flatcar-stable-4081.2.0-kube-v1.30.1.img.raw1    4096   266239   262144  128M EFI System
flatcar-stable-4081.2.0-kube-v1.30.1.img.raw2  266240   270335     4096    2M BIOS boot
flatcar-stable-4081.2.0-kube-v1.30.1.img.raw3  270336  2367487  2097152    1G unknown
flatcar-stable-4081.2.0-kube-v1.30.1.img.raw4 2367488  4464639  2097152    1G unknown
flatcar-stable-4081.2.0-kube-v1.30.1.img.raw6 4464640  4726783   262144  128M Linux filesystem
flatcar-stable-4081.2.0-kube-v1.30.1.img.raw7 4726784  4857855   131072   64M unknown
flatcar-stable-4081.2.0-kube-v1.30.1.img.raw9 4857856 41943006 37085151 17.7G unknown

We apparently have Nova configured to convert qcow2 images to raw before
booting them, which we can also see in the logs:

Dec 04 11:01:37 capo-e2e-controller.c.k8s-infra-e2e-boskos-107.internal
nova-compute[114399]: ESC[00;32mDEBUG nova.virt.images [ESC[01;36mNone
req-993c42cb-8da1-4cc5-83fd-1c16c08cbc13 ESC[00;36mdemo demoESC[00;32m]
ESC[01;35mESC[00;32m945136cb-6cc4-4e09-a785-50eaa79e2b10 was qcow2,
converting to rawESC[00m ESC[00;33m{{(pid=114399) fetch_to_raw
/opt/stack/nova/nova/virt/images.py:254}}ESC[00mESC[00m


Using a patch to oslo.utils from Stephen Finucane and adding some extra print statements of my own, it's clear that we're failing here:

https://github.com/openstack/oslo.utils/blob/79f5ec658e2fee8ab46201a71faaff8d3b67a690/oslo_utils/imageutils/format_inspector.py#L1273-L1274

> ./venv/bin/python ./oslo_utils/imageutils/format_inspector.py /tmp/flatcar-stable-4081.2.0-kube-v1.30.1.img.raw
inspecting file: /tmp/flatcar-stable-4081.2.0-kube-v1.30.1.raw
detected file format: gpt
running safety checks...
i: 0, ostype: 12
i: 1, ostype: 238
i: 2, ostype: 0
i: 3, ostype: 0
valid_partions: [0, 1]
Safety check mbr on gpt failed because GPT MBR defines invalid extra partitions
FAILED! Safety checks failed: mbr

1/1 failed

This code expects there to be exactly one partition with a non-zero
partition type, and that this partition must be the first one. In this
image, both of the first 2 partitions have a non-zero partition type.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2091114

Title:
  Nova validation checks checks reject valid UEFI image

Status in OpenStack Compute (nova):
  New

Bug description:
  This relates specifically to this image:
  https://storage.googleapis.com/artifacts.k8s-staging-capi-openstack.appspot.com/test/flatcar/flatcar-stable-4081.2.0-kube-v1.30.1.img

  However, the problem should be easy enough to understand just from the
  description here without downloading it.

  When attempting to boot the image in 2024.2 devstack we see the
  following failure:

  Dec 04 11:01:53
  capo-e2e-controller.c.k8s-infra-e2e-boskos-107.internal nova-
  compute[114399]: ESC[01;33mWARNING
  oslo_utils.imageutils.format_inspector [ESC[01;36mNone
  req-993c42cb-8da1-4cc5-83fd-1c16c08cbc13 ESC[00;36mdemo
  demoESC[01;33m] ESC[01;35mESC[01;33mSafety check mbr on gpt failed
  because GPT MBR defines invalid extra partitionsESC[00m:
  oslo_utils.imageutils.format_inspector.SafetyViolation: GPT MBR
  defines invalid extra partitionsESC[00m

  There is an associated stack trace and the server enters the ERROR
  state.

  This is a QCOW2 image. After downloading it I can manually convert it
  to raw to inspect its partition table:

  > qemu-img convert -f qcow2 flatcar-stable-4081.2.0-kube-v1.30.1.img -O raw flatcar-stable-4081.2.0-kube-v1.30.1.img.raw
  > fdisk -l flatcar-stable-4081.2.0-kube-v1.30.1.img.raw
  Disk flatcar-stable-4081.2.0-kube-v1.30.1.img.raw: 20 GiB, 21474836480 bytes, 41943040 sectors
  Units: sectors of 1 * 512 = 512 bytes
  Sector size (logical/physical): 512 bytes / 512 bytes
  I/O size (minimum/optimal): 512 bytes / 512 bytes
  Disklabel type: gpt
  Disk identifier: D814FAF6-AD0A-4FC1-8DE9-236755D902E5

  Device                                          Start      End  Sectors  Size Type
  flatcar-stable-4081.2.0-kube-v1.30.1.img.raw1    4096   266239   262144  128M EFI System
  flatcar-stable-4081.2.0-kube-v1.30.1.img.raw2  266240   270335     4096    2M BIOS boot
  flatcar-stable-4081.2.0-kube-v1.30.1.img.raw3  270336  2367487  2097152    1G unknown
  flatcar-stable-4081.2.0-kube-v1.30.1.img.raw4 2367488  4464639  2097152    1G unknown
  flatcar-stable-4081.2.0-kube-v1.30.1.img.raw6 4464640  4726783   262144  128M Linux filesystem
  flatcar-stable-4081.2.0-kube-v1.30.1.img.raw7 4726784  4857855   131072   64M unknown
  flatcar-stable-4081.2.0-kube-v1.30.1.img.raw9 4857856 41943006 37085151 17.7G unknown

  We apparently have Nova configured to convert qcow2 images to raw
  before booting them, which we can also see in the logs:

  Dec 04 11:01:37
  capo-e2e-controller.c.k8s-infra-e2e-boskos-107.internal nova-
  compute[114399]: ESC[00;32mDEBUG nova.virt.images [ESC[01;36mNone
  req-993c42cb-8da1-4cc5-83fd-1c16c08cbc13 ESC[00;36mdemo
  demoESC[00;32m]
  ESC[01;35mESC[00;32m945136cb-6cc4-4e09-a785-50eaa79e2b10 was qcow2,
  converting to rawESC[00m ESC[00;33m{{(pid=114399) fetch_to_raw
  /opt/stack/nova/nova/virt/images.py:254}}ESC[00mESC[00m

  
  Using a patch to oslo.utils from Stephen Finucane and adding some extra print statements of my own, it's clear that we're failing here:

  https://github.com/openstack/oslo.utils/blob/79f5ec658e2fee8ab46201a71faaff8d3b67a690/oslo_utils/imageutils/format_inspector.py#L1273-L1274

  > ./venv/bin/python ./oslo_utils/imageutils/format_inspector.py /tmp/flatcar-stable-4081.2.0-kube-v1.30.1.img.raw
  inspecting file: /tmp/flatcar-stable-4081.2.0-kube-v1.30.1.raw
  detected file format: gpt
  running safety checks...
  i: 0, ostype: 12
  i: 1, ostype: 238
  i: 2, ostype: 0
  i: 3, ostype: 0
  valid_partions: [0, 1]
  Safety check mbr on gpt failed because GPT MBR defines invalid extra partitions
  FAILED! Safety checks failed: mbr

  1/1 failed

  This code expects there to be exactly one partition with a non-zero
  partition type, and that this partition must be the first one. In this
  image, both of the first 2 partitions have a non-zero partition type.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2091114/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next