yahoo-eng-team team mailing list archive

[Jeffrey Zhang <2056513@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

reproduce

* launch a 2 cpu vm with multi queue enabled, also configure a max vcpu
as bellow

<vcpu placement='static' current='2'>4</vcpu>

* increase the vcpu from 2 to 4

virsh setvcpus XXX 4

* migration the node

nova live-migration XXX

will get following log in nova-compute.log

2024-03-08 11:32:05.795 8 INFO nova.compute.manager [req-1c0a7415-d81b-480a-a1f9-06b9afdb2b55 - - - - -] [None] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] During sync_power_state the instance has a pending task (migrating). Skip.
2024-03-08 11:32:06.673 8 INFO nova.compute.manager [req-1c0a7415-d81b-480a-a1f9-06b9afdb2b55 - - - - -] [None] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] VM Resumed (Lifecycle Event)
2024-03-08 11:32:06.843 8 ERROR nova.virt.libvirt.driver [-] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] Live Migration failure: internal error: qemu unexpectedly closed the monitor: 2024-03-08T03:32:03.383294Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: CPU 2 [socket-id: 2, core-id: 0, thread-id: 0], CPU 3 [socket-id: 3, core-id: 0, thread-id: 0]
2024-03-08T03:32:03.383370Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
2024-03-08T03:32:05.584824Z qemu-kvm: get_pci_config_device: Bad config data: i=0x9a read: 5 device: 9 cmask: ff wmask: 0 w1cmask:0
2024-03-08T03:32:05.584971Z qemu-kvm: Failed to load PCIDevice:config
2024-03-08T03:32:05.585014Z qemu-kvm: Failed to load virtio-net:virtio
2024-03-08T03:32:05.585072Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-net'
2024-03-08T03:32:05.587982Z qemu-kvm: load of migration failed: Invalid argument: libvirtError: internal error: qemu unexpectedly closed the monitor: 2024-03-08T03:32:03.383294Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: CPU 2 [socket-id: 2, core-id: 0, thread-id: 0], CPU 3 [socket-id: 3, core-id: 0, thread-id: 0]
2024-03-08 11:32:06.863 8 INFO nova.compute.manager [req-1c0a7415-d81b-480a-a1f9-06b9afdb2b55 - - - - -] [None] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] During sync_power_state the instance has a pending task (migrating). Skip.
2024-03-08 11:32:07.019 8 ERROR nova.virt.libvirt.driver [-] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] Migration operation has aborted


# root cause

in the dst node, nova will configure the nic multi queue from 2 to 4,
which is not the same with original  . So qemu failed to start

# fix

use original queue count rather than generate at dst node.

** Affects: nova
     Importance: Undecided
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2056513

Title:
  live migration will failed if the vm enabled nic multi queue and vcpu
  is live resize to a max value

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  reproduce

  * launch a 2 cpu vm with multi queue enabled, also configure a max
  vcpu as bellow

  <vcpu placement='static' current='2'>4</vcpu>

  * increase the vcpu from 2 to 4

  virsh setvcpus XXX 4

  * migration the node

  nova live-migration XXX

  will get following log in nova-compute.log

  2024-03-08 11:32:05.795 8 INFO nova.compute.manager [req-1c0a7415-d81b-480a-a1f9-06b9afdb2b55 - - - - -] [None] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] During sync_power_state the instance has a pending task (migrating). Skip.
  2024-03-08 11:32:06.673 8 INFO nova.compute.manager [req-1c0a7415-d81b-480a-a1f9-06b9afdb2b55 - - - - -] [None] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] VM Resumed (Lifecycle Event)
  2024-03-08 11:32:06.843 8 ERROR nova.virt.libvirt.driver [-] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] Live Migration failure: internal error: qemu unexpectedly closed the monitor: 2024-03-08T03:32:03.383294Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: CPU 2 [socket-id: 2, core-id: 0, thread-id: 0], CPU 3 [socket-id: 3, core-id: 0, thread-id: 0]
  2024-03-08T03:32:03.383370Z qemu-kvm: warning: All CPU(s) up to maxcpus should be described in NUMA config, ability to start up with partial NUMA mappings is obsoleted and will be removed in future
  2024-03-08T03:32:05.584824Z qemu-kvm: get_pci_config_device: Bad config data: i=0x9a read: 5 device: 9 cmask: ff wmask: 0 w1cmask:0
  2024-03-08T03:32:05.584971Z qemu-kvm: Failed to load PCIDevice:config
  2024-03-08T03:32:05.585014Z qemu-kvm: Failed to load virtio-net:virtio
  2024-03-08T03:32:05.585072Z qemu-kvm: error while loading state for instance 0x0 of device '0000:00:03.0/virtio-net'
  2024-03-08T03:32:05.587982Z qemu-kvm: load of migration failed: Invalid argument: libvirtError: internal error: qemu unexpectedly closed the monitor: 2024-03-08T03:32:03.383294Z qemu-kvm: warning: CPU(s) not present in any NUMA nodes: CPU 2 [socket-id: 2, core-id: 0, thread-id: 0], CPU 3 [socket-id: 3, core-id: 0, thread-id: 0]
  2024-03-08 11:32:06.863 8 INFO nova.compute.manager [req-1c0a7415-d81b-480a-a1f9-06b9afdb2b55 - - - - -] [None] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] During sync_power_state the instance has a pending task (migrating). Skip.
  2024-03-08 11:32:07.019 8 ERROR nova.virt.libvirt.driver [-] [instance: 5a317564-9e87-4313-a8a3-c661434c5bc1] Migration operation has aborted

  
  # root cause

  in the dst node, nova will configure the nic multi queue from 2 to 4,
  which is not the same with original  . So qemu failed to start

  # fix

  use original queue count rather than generate at dst node.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2056513/+subscriptions