yahoo-eng-team team mailing list archive

[Dan Smith <2079850@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

When configured to format ephemerals as vfat, we get this failure:

Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.358 2 DEBUG oslo_utils.imageutils.format_inspector [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] Format inspector failed, aborting: Signature KDMV not found: b'\xebX\x90m' _process_chunk /usr/lib/python3.9/site-packages/oslo_utils/imageutils/format_inspector.py:1302
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.365 2 DEBUG oslo_utils.imageutils.format_inspector [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] Format inspector failed, aborting: Region signature not found at 30000 _process_chunk /usr/lib/python3.9/site-packages/oslo_utils/imageutils/format_inspector.py:1302
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.366 2 WARNING oslo_utils.imageutils.format_inspector [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] Safety check mbr on gpt failed because GPT MBR has no partitions defined: oslo_utils.imageutils.format_inspector.SafetyViolation: GPT MBR has no partitions defined
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.366 2 WARNING nova.virt.libvirt.imagebackend [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] Base image /var/lib/nova/instances/_base/ephemeral_1_0706d66 failed safety check: Safety checks failed: mbr: oslo_utils.imageutils.format_inspector.SafetyCheckFailed: Safety checks failed: mbr
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] Instance failed to spawn: nova.exception.InvalidDiskInfo: Disk info file is invalid: Base image failed safety check
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] Traceback (most recent call last):
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/imagebackend.py", line 685, in create_image
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     inspector.safety_check()
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/oslo_utils/imageutils/format_inspector.py", line 430, in safety_check
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     raise SafetyCheckFailed(failures)
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] oslo_utils.imageutils.format_inspector.SafetyCheckFailed: Safety checks failed: mbr
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] 
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] During handling of the above exception, another exception occurred:
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] 
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] Traceback (most recent call last):
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2894, in _build_resources
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     yield resources
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2641, in _build_and_run_instance
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     self.driver.spawn(context, instance, image_meta,
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4734, in spawn
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     created_instance_dir, created_disks = self._create_image(
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5198, in _create_image
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     disk_image.cache(fetch_func=fn,
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/imagebackend.py", line 301, in cache
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     self.create_image(fetch_func_sync, base, size,
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/imagebackend.py", line 691, in create_image
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     raise exception.InvalidDiskInfo(
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] nova.exception.InvalidDiskInfo: Disk info file is invalid: Base image failed safety check
Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] 

There are a number of problems here:

1. Nova shouldn't need to inspect blank disks it has created by itself, but because of the nature of the imagebackend, that code can't tell the difference
2. FAT actually starts with a VBR which looks like an MBR because it's the same data structure and signature, but without any partition entries. This leftover from the 1980s is inconvenient for us, but the format_inspector should try to avoid detecting these FAT images as GPT/MBR

** Affects: nova
     Importance: Undecided
         Status: New

** Affects: oslo.utils
     Importance: Undecided
         Status: New

** Also affects: oslo.utils
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2079850

Title:
  Ephemeral with vfat format fails inspection

Status in OpenStack Compute (nova):
  New
Status in oslo.utils:
  New

Bug description:
  When configured to format ephemerals as vfat, we get this failure:

  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.358 2 DEBUG oslo_utils.imageutils.format_inspector [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] Format inspector failed, aborting: Signature KDMV not found: b'\xebX\x90m' _process_chunk /usr/lib/python3.9/site-packages/oslo_utils/imageutils/format_inspector.py:1302
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.365 2 DEBUG oslo_utils.imageutils.format_inspector [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] Format inspector failed, aborting: Region signature not found at 30000 _process_chunk /usr/lib/python3.9/site-packages/oslo_utils/imageutils/format_inspector.py:1302
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.366 2 WARNING oslo_utils.imageutils.format_inspector [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] Safety check mbr on gpt failed because GPT MBR has no partitions defined: oslo_utils.imageutils.format_inspector.SafetyViolation: GPT MBR has no partitions defined
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.366 2 WARNING nova.virt.libvirt.imagebackend [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] Base image /var/lib/nova/instances/_base/ephemeral_1_0706d66 failed safety check: Safety checks failed: mbr: oslo_utils.imageutils.format_inspector.SafetyCheckFailed: Safety checks failed: mbr
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [None req-fcf3a278-3417-4a6d-8b10-66e91ca1677d 60ed4d3e522640b6ad19633b28c5b5bb ae43aec9c3c242a785c8256abdda1747 - - default default] [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] Instance failed to spawn: nova.exception.InvalidDiskInfo: Disk info file is invalid: Base image failed safety check
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] Traceback (most recent call last):
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/imagebackend.py", line 685, in create_image
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     inspector.safety_check()
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/oslo_utils/imageutils/format_inspector.py", line 430, in safety_check
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     raise SafetyCheckFailed(failures)
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] oslo_utils.imageutils.format_inspector.SafetyCheckFailed: Safety checks failed: mbr
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] 
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] During handling of the above exception, another exception occurred:
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] 
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] Traceback (most recent call last):
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2894, in _build_resources
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     yield resources
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2641, in _build_and_run_instance
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     self.driver.spawn(context, instance, image_meta,
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4734, in spawn
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     created_instance_dir, created_disks = self._create_image(
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5198, in _create_image
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     disk_image.cache(fetch_func=fn,
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/imagebackend.py", line 301, in cache
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     self.create_image(fetch_func_sync, base, size,
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/imagebackend.py", line 691, in create_image
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a]     raise exception.InvalidDiskInfo(
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] nova.exception.InvalidDiskInfo: Disk info file is invalid: Base image failed safety check
  Sep 03 17:34:28 compute-2 nova_compute[133243]: 2024-09-03 17:34:28.367 2 ERROR nova.compute.manager [instance: 263ccd01-10b1-46a6-9f81-a6fc27c7177a] 

  There are a number of problems here:

  1. Nova shouldn't need to inspect blank disks it has created by itself, but because of the nature of the imagebackend, that code can't tell the difference
  2. FAT actually starts with a VBR which looks like an MBR because it's the same data structure and signature, but without any partition entries. This leftover from the 1980s is inconvenient for us, but the format_inspector should try to avoid detecting these FAT images as GPT/MBR

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2079850/+subscriptions