yahoo-eng-team team mailing list archive

[Pavlo Shchelokovskyy <2038898@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Discovered in a environment that was configured with

[libvirt]
images_type = raw

only, the other relevant options were at their defaults (use_cow_images
= True, force_raw_images = False).

Symptom - the instances were non-responsive and non running after cold migration (e.g. no console log at all), live migration works fine.
Workaround - setting use_cow_images=False and force_raw_images=True solved the problem.

Reproduction on a current multinode devstack:

1. Configure computes as described above - set [libvirt]images_type = raw, leave the rest per default devstack / nova settings.
2. Create a raw image in Glance.
3. Boot an instance from that raw image.
4. Inspect the image on the file system - the image is in fact raw.
5. Cold-migrate the server.
6. Migration finishes successfully, instance is reported as up and running on the new host - but in fact it has completely failed to start (not accessible, no console log, nothing).
7. If you check the image file nova uses on the new compute - it is now qcow2, not raw.
8. But the libvirt XML of the instance still defines the disk as raw!

Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:   <devices>
Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:     <disk type="file" device="disk">
Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:       <driver name="qemu" type="raw" cache="none"/>
Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:       <source file="/opt/stack/data/nova/instances/22749d77-83a1-4ae9-ade8-7bd9548406cd/disk"/>
Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:       <target dev="vda" bus="virtio"/>
Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:     </disk>

Stopping the instance and manually converting the disk back to raw
allows instance to start properly.

I tracked it down to this place in finish_migration method:

https://opendev.org/openstack/nova/src/branch/stable/2023.2/nova/virt/libvirt/driver.py#L11739

            if (disk_name != 'disk.config' and
                        info['type'] == 'raw' and CONF.use_cow_images):
                self._disk_raw_to_qcow2(info['path'])

Effectively, nova changes disk type but not changing the XML
appropriately to reflect the actual new disk format, and thus the
instance fails to start.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2038898

Title:
  image format change during migration is not reflected in libvirt XML

Status in OpenStack Compute (nova):
  New

Bug description:
  Discovered in a environment that was configured with

  [libvirt]
  images_type = raw

  only, the other relevant options were at their defaults
  (use_cow_images = True, force_raw_images = False).

  Symptom - the instances were non-responsive and non running after cold migration (e.g. no console log at all), live migration works fine.
  Workaround - setting use_cow_images=False and force_raw_images=True solved the problem.

  Reproduction on a current multinode devstack:

  1. Configure computes as described above - set [libvirt]images_type = raw, leave the rest per default devstack / nova settings.
  2. Create a raw image in Glance.
  3. Boot an instance from that raw image.
  4. Inspect the image on the file system - the image is in fact raw.
  5. Cold-migrate the server.
  6. Migration finishes successfully, instance is reported as up and running on the new host - but in fact it has completely failed to start (not accessible, no console log, nothing).
  7. If you check the image file nova uses on the new compute - it is now qcow2, not raw.
  8. But the libvirt XML of the instance still defines the disk as raw!

  Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:   <devices>
  Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:     <disk type="file" device="disk">
  Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:       <driver name="qemu" type="raw" cache="none"/>
  Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:       <source file="/opt/stack/data/nova/instances/22749d77-83a1-4ae9-ade8-7bd9548406cd/disk"/>
  Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:       <target dev="vda" bus="virtio"/>
  Oct 09 12:15:35 pshchelo-devstack-jammy nova-compute[427994]:     </disk>

  Stopping the instance and manually converting the disk back to raw
  allows instance to start properly.

  I tracked it down to this place in finish_migration method:

  https://opendev.org/openstack/nova/src/branch/stable/2023.2/nova/virt/libvirt/driver.py#L11739

              if (disk_name != 'disk.config' and
                          info['type'] == 'raw' and CONF.use_cow_images):
                  self._disk_raw_to_qcow2(info['path'])

  Effectively, nova changes disk type but not changing the XML
  appropriately to reflect the actual new disk format, and thus the
  instance fails to start.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2038898/+subscriptions