← Back to team overview

yahoo-eng-team team mailing list archive

[Bug 1946298] Re: live-migration fails when option rom size is different in guest's memory

 

We discussed this bug report during the latest Asian-friendly Nova
meeting [1] and we agreed on the fact this report is about asking Nova
to support iPXE, as for the moment Nova doesn't really expose it for the
moment, even if libvirtd does this.

Please provide a blueprint for explaining your needs and then we will
discuss on it.


[1] https://meetings.opendev.org/meetings/nova_extra/2021/nova_extra.2021-10-07-08.04.log.html#l-59


** Changed in: nova
       Status: New => Invalid

** Changed in: nova
   Importance: Undecided => Wishlist

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1946298

Title:
  live-migration fails when option rom size is different in guest's
  memory

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Description
  ===========
  This problem is found when doing live-migration accross nova versions,
  especially ipxe versions depended on libvirt. If an instance is attached to
  interface, option rom is loaded into guest's rom. When doing live migration,
  qemu will check rom size and try to resize resizable memory region. However,
  option rom is not resizable. Once the destination node found option rom size
  changed when loaded to memory, an exception will occur and stop the migration
  process.

  Steps to reproduce
  ==================
  A simple way to reproduce:
  * Prepare two nova-compute node, which can be the same version
  * Create an instance on Node A, and attach an interface to it
  * Check which ipxe rom is loaded into memory by its model type.
    For example, if an interface is defined with `<model type='virtio'/>`, then
    `/usr/lib/ipxe/qemu/efi-virtio.rom` is loaded to rom on ubuntu x86 system.
  * Change the rom's virtual size on the destination Node B. 
  Simply `echo "hello" > /usr/lib/ipxe/qemu/efi-virtio.rom`
  The virtual size is the max length when rom is loaded to guest's memory, which
  is exponential times of 2. We can use the following command to get rom's 
  virtual size.
    `virsh qemu-monitor-command <domain> --hmp 'info ramblock'` 
  * Do live-migration

    `nova live-migration --block-migrate cirros1 cmp02`

  Expected result
  ===============
  Normally, if the rom's virtual size is not changed, migration will succeed.

  Actual result
  =============
  After the execution of the steps above, the live-migration will fail with 
  error.

  Environment
  ===========
  Nova version:
  $ dpkg -l | grep nova
  ii  nova-common                            2:21.2.1-0ubuntu1     all
  ii  nova-compute                           2:21.2.1-0ubuntu1     all
  ii  nova-compute-kvm                       2:21.2.1-0ubuntu1     all
  ii  nova-compute-libvirt                   2:21.2.1-0ubuntu1     all
  ii  python3-nova                           2:21.2.1-0ubuntu1     all
  ii  python3-novaclient                     2:17.0.0-0ubuntu1     all

  Hypervisor type: libvirt
  $ dpkg -l | grep libvirt
  ii  libvirt-clients                        6.0.0-0ubuntu8.13     amd64
  ii  libvirt-daemon                         6.0.0-0ubuntu8.13     amd64
  ii  libvirt-daemon-driver-qemu             6.0.0-0ubuntu8.13     amd64
  ii  libvirt-daemon-driver-storage-rbd      6.0.0-0ubuntu8.13     amd64
  ii  libvirt-daemon-system                  6.0.0-0ubuntu8.13     amd64
  ii  libvirt-daemon-system-systemd          6.0.0-0ubuntu8.13     amd64
  ii  libvirt0:amd64                         6.0.0-0ubuntu8.13     amd64
  ii  nova-compute-libvirt                   2:21.2.1-0ubuntu1     all
  ii  python3-libvirt                        6.1.0-1               amd64

  Networking type: Neutron with OpenVSwitch
  $ dpkg -l | grep neutron
  ii  neutron-common                         2:16.4.0-0ubuntu3     all
  ii  neutron-openvswitch-agent              2:16.4.0-0ubuntu3     all
  ii  python3-neutron                        2:16.4.0-0ubuntu3     all
  ii  python3-neutron-lib                    2.3.0-0ubuntu1        all
  ii  python3-neutronclient                  1:7.1.1-0ubuntu1      all

  Logs & Configs
  ==============
  ```text
  2021-09-22 10:10:31.451 35235 ERROR nova.virt.libvirt.driver [-] [instance: 6d91c241-75b8-4067-8874-c64970b87f6a] Migration operation has aborted
  2021-09-22 10:10:31.644 35235 INFO nova.compute.manager [-] [instance: 6d91c241-75b8-4067-8874-c64970b87f6a] Swapping old allocation on dict_keys(['61b9a486-f53e-4b70-b54c-0db29f8ff978']) held by migration f5308871-0e91-48b0-8a68-a7d66239b3bd for instance
  2021-09-22 10:10:31.671 35235 ERROR nova.virt.libvirt.driver [-] [instance: 6d91c241-75b8-4067-8874-c64970b87f6a] Live Migration failure: internal error: qemu unexpectedly closed the monitor: 2021-09-22T02:10:31.450377Z qemu-system-x86_64: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x1000 in != 0x80000: Invalid argument
  2021-09-22T02:10:31.450414Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
  2021-09-22T02:10:31.452282Z qemu-system-x86_64: load of migration failed: Invalid argument: libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2021-09-22T02:10:31.450377Z qemu-system-x86_64: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x1000 in != 0x80000: Invalid argument
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1946298/+subscriptions



References