yahoo-eng-team team mailing list archive

[OpenStack Infra <2071734@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/924731
Committed: https://opendev.org/openstack/nova/commit/8b4c522f6699514e7d1f20ac25cf426af6ea588f
Submitter: "Zuul (22348)"
Branch:    master

commit 8b4c522f6699514e7d1f20ac25cf426af6ea588f
Author: Dan Smith <dansmith@xxxxxxxxxx>
Date:   Wed Jul 10 14:23:33 2024 +0100

    Change force_format strategy to catch mismatches
    
    When we moved the qemu-img command in fetch_to_raw() to force the
    format to what we expect, we lost the ability to identify and react
    to situations where qemu-img detected a file as a format that is not
    supported by us (i.e. identfied and safety-checked by
    format_inspector). In the case of some of the other VMDK variants
    that we don't support, we need to be sure to catch any case where
    qemu-img thinks it's something other than raw when we think it is,
    which will be the case for those formats we don't support.
    
    Note this also moves us from explicitly using the format_inspector
    that we're told by glance is appropriate, to using our own detection.
    We assert that we agree with glance and as above, qemu agrees with
    us. This helps us avoid cases where the uploader lies about the
    image format, causing us to not run the appropriate safety check.
    AMI formats are a liability here since we have a very hard time
    asserting what they are and what they will be detected as later in
    the pipeline, so there is still special-casing for those.
    
    Closes-Bug: #2071734
    Change-Id: I4b792c5bc959a904854c21565682ed3a687baa1a


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2071734

Title:
  [OSSA-2024-002] Incomplete file access fix and regression for QCOW2
  backing files and VMDK flat descriptors (CVE-2024-40767)

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Security Advisory:
  Fix Released

Bug description:
  When fixing bug #2059809, a regression of the previous bug #1996188
  has been introduced.

  TLDR: nova is allowing back VMDK with wrong types and QCOW with
  backing files.

  Long:

  The following steps were used to reproduce on a Bobcat (2023.2)
  OpenStack + nova patches for bug #2059809 (not yet merge when writing
  this report)

  Create a vmdk file:
  $ qemu-img create -f vmdk disk-vmdk.vmdk 1M -o subformat=monolithicFlat
  $ sed -i -r 's|disk-vmdk-flat.vmdk|/etc/hosts|' disk-vmdk.vmdk

  Create a faulty qcow image:
  $ qemu-img create -f qcow2 -F raw -b /etc/hosts disk-bf.qcow2 1M

  Upload both images as raw (the default)
  $ for i in disk-bf.qcow2 disk-vmdk.vmdk ; do openstack image create --file $i $i ; done

  Boot an instance from those images:
  $ openstack server create --flavor small --image disk-vmdk.vmdk --net public disk-vmdk.vmdk
  $ openstack server create --flavor small --image disk-bf.qcow2 --net public disk-bf.qcow2

  Results
  =======
  VMDK monolithicFlat
  -------------------

  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2615, in _build_and_run_instance
      self.driver.spawn(context, instance, image_meta,
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4388, in spawn
      created_instance_dir, created_disks = self._create_image(
                                            ^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4790, in _create_image
      created_disks = self._create_and_inject_local_root(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4921, in _create_and_inject_local_root
      self._try_fetch_image_cache(backend, fetch_func, context,
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10940, in _try_fetch_image_cache
      image.cache(fetch_func=fetch_func,
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 288, in cache
      self.create_image(fetch_func_sync, base, size,
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 615, in create_image
      copy_raw_image(base, self.path, size)
    File "/usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py", line 414, in inner
      return f(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 590, in copy_raw_image
      self.resize_image(size)
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/imagebackend.py", line 621, in resize_image
      disk.extend(image, size)
    File "/usr/lib/python3/dist-packages/nova/virt/disk/api.py", line 128, in extend
      processutils.execute('qemu-img', 'resize', image.path, size)
    File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py", line 438, in execute
      raise ProcessExecutionError(exit_code=_returncode,
  oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
  Command: qemu-img resize /var/lib/nova/instances/07b5907b-5efc-4fdf-8b15-4b47a820c2f8/disk 2147483648
  Exit code: 1
  Stdout: ''
  Stderr: "qemu-img: Could not open '/var/lib/nova/instances/07b5907b-5efc-4fdf-8b15-4b47a820c2f8/disk': Could not open '/etc/hosts': Permission denied\n"

  Qemu tried to read /etc/hosts. My system permissions prevented it, but
  nova did nothing about it: wrong.

  QEMU Backing File
  -----------------
  Traceback (most recent call last):
    File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 2615, in _build_and_run_instance
      self.driver.spawn(context, instance, image_meta,
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 4415, in spawn
      self._create_guest_with_network(
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7785, in _create_guest_with_network
      with excutils.save_and_reraise_exception():
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7763, in _create_guest_with_network
      guest = self._create_guest(
              ^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 7702, in _create_guest
      guest.launch(pause=pause)
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 167, in launch
      with excutils.save_and_reraise_exception():
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 165, in launch
      return self._domain.createWithFlags(flags)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in proxy_call
      rv = execute(f, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
      six.reraise(c, e, tb)
    File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
      raise value
    File "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
      rv = meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
    File "/usr/lib/python3/dist-packages/libvirt.py", line 1409, in createWithFlags
      raise libvirtError('virDomainCreateWithFlags() failed')
  libvirt.libvirtError: internal error: cannot update AppArmor profile 'libvirt-6bd32822-2454-402a-9617-6ec66e0090f4'

  In libvirtd journalctl:

  Jul 02 20:22:44 compute-1 libvirtd[959438]: internal error: Child
  process (LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/virt-aa-helper
  -r -u libvirt-6bd32822-2454-402a-9617-6ec66e0090f4 -F /dev/net/tun)
  unexpected exit status 1: virt-aa-helper: error: /etc/hosts

  Here it's apparmor that prevented the boot, but nova should have
  catched it: wrong

  Expected results
  ----------------

  Nova should raise an exception like it does previously.
  E.G. for VMDK: nova.exception.ImageUnacceptable: Image xyz is unacceptable: Invalid VMDK create-type specified

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2071734/+subscriptions