← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #84861

[Bug 1910331] Re: Live migration CPU pre check failed

  Thread Previous • Date Previous • • Thread Next 
*** This bug is a duplicate of bug 1898715 ***
    https://bugs.launchpad.net/bugs/1898715

It seems like a duplicate of
https://bugs.launchpad.net/nova/+bug/1898715 for which the patch merged
about a months ago https://review.opendev.org/c/openstack/nova/+/758761

I'm marking this as duplicate. Please remove the duplication tag if you
think this is a separate issue.

** This bug has been marked a duplicate of bug 1898715
   Live migration fails despite matching CPUs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1910331

Title:
  Live migration CPU pre check failed

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  During live-migration with --block-migrate enable, CPU check always fail even both source host and destination host are identical in term of hardware and software version. 

  Steps to reproduce
  ==================
  # nova live-migration --block-migrate instance computenode2

  After a bit, check on computenode2, I can see the error in the log
  below

  Expected result
  ===============
  instance got moved to computenode2

  Actual result
  =============
  instance is still on its old compute node

  Environment
  ===========
  1. Openstack version, OS version and libvirt version
  #  rpm -qa | egrep "nova|kvm|libvirt" | sort
  libvirt-bash-completion-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-client-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-interface-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-network-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-nodedev-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-nwfilter-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-qemu-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-secret-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-core-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-disk-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-gluster-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-iscsi-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-iscsi-direct-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-logical-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-mpath-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-rbd-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-driver-storage-scsi-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-daemon-kvm-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  libvirt-libs-6.0.0-28.module_el8.3.0+555+a55c8938.x86_64
  openstack-nova-common-20.4.1-1.el8.noarch
  openstack-nova-compute-20.4.1-1.el8.noarch
  python3-libvirt-6.0.0-1.module_el8.3.0+555+a55c8938.x86_64
  python3-nova-20.4.1-1.el8.noarch
  python3-novaclient-15.1.1-1.el8.noarch
  qemu-kvm-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
  qemu-kvm-block-curl-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
  qemu-kvm-block-gluster-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
  qemu-kvm-block-iscsi-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
  qemu-kvm-block-rbd-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
  qemu-kvm-block-ssh-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
  qemu-kvm-common-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64
  qemu-kvm-core-4.2.0-34.module_el8.3.0+555+a55c8938.x86_64

  # cat /etc/redhat-release
  CentOS Linux release 8.3.2011

  # uname -a
  Linux computenode2 4.18.0-240.1.1.el8_3.x86_64 #1 SMP Thu Nov 19 17:20:08 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

  
  2. Which hypervisor did you use?
  KVM

  3. Which networking type did you use?
  Neutron with OpenVSwitch

  
  Logs & Configs
  ==============

  - On computende2 I can see this error:

  2021-01-06 10:57:45.795 6992 INFO nova.virt.libvirt.driver [req-34f4dc34-dcb1-4632-813b-caf9f1a47439 7584ec4fa0fc45ddb25c2603357a7912 b5608fc845894e68abe34703c801b6e6 - default default] Instance launched has CPU info
  : {"arch": "x86_64", "model": "Cascadelake-Server-noTSX", "vendor": "Intel", "topology": {"cells": 2, "sockets": 1, "cores": 22, "threads": 2}, "features": ["lm", "pdpe1gb", "arch-capabilities", "smx", "lahf_lm", "rt
  m", "avx", "fma", "est", "ss", "vmx", "avx512dq", "fsgsbase", "ibrs-all", "rdtscp", "taa-no", "mmx", "pclmuldq", "xtpr", "clflushopt", "spec-ctrl", "mca", "dtes64", "mds-no", "apic", "pcid", "fpu", "monitor", "cx16",
   "stibp", "avx512f", "adx", "md-clear", "tm2", "pse", "hle", "invpcid", "sse2", "xsaveopt", "xsavec", "cx8", "popcnt", "pse36", "tsc-deadline", "avx512-bf16", "ht", "tm", "arat", "avx512vnni", "de", "pni", "mpx", "ts
  x-ctrl", "3dnowprefetch", "f16c", "smap", "sse4.2", "avx512bw", "xsave", "acpi", "pge", "ds_cpl", "bmi1", "ssbd", "pku", "avx512cd", "smep", "tsc_adjust", "aes", "clflush", "pdcm", "cmov", "fxsr", "rdseed", "avx2", "
  ds", "pbe", "rdctl-no", "bmi2", "sse", "sse4.1", "abm", "pat", "mce", "pae", "invtsc", "tsc", "x2apic", "xgetbv1", "msr", "clwb", "sep", "skip-l1dfl-vmentry", "vme", "pschange-mc-no", "movbe", "syscall", "ssse3", "nx
  ", "xsaves", "dca", "mtrr", "avx512vl", "rdrand", "erms", "intel-pt"]}
  2021-01-06 10:57:45.797 6992 ERROR nova.virt.libvirt.driver [req-34f4dc34-dcb1-4632-813b-caf9f1a47439 7584ec4fa0fc45ddb25c2603357a7912 b5608fc845894e68abe34703c801b6e6 - default default] CPU doesn't have compatibility.

  0

  Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server [req-34f4dc34-dcb1-4632-813b-caf9f1a47439 7584ec4fa0fc45ddb25c2603357a7912 b5608fc845894e68abe34703c801b6e6 - default default] Exception during message handling: nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.

  0

  Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 79, in wrapped
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary, tb)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     raise value
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/exception_wrapper.py", line 69, in wrapped
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/utils.py", line 1372, in decorated_function
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 219, in decorated_function
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     self.force_reraise()
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     raise value
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 207, in decorated_function
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/compute/manager.py", line 6789, in check_can_live_migrate_destination
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     block_migration, disk_over_commit)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8114, in check_can_live_migrate_destination
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     self._compare_cpu(None, source_cpu_info, instance)
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 8439, in _compare_cpu
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server     raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u})
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server 0
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
  2021-01-06 10:57:45.835 6992 ERROR oslo_messaging.rpc.server

  
  - nova config on each compute node:

  # egrep -v "^#|^$" /etc/nova/nova.conf
  [libvirt]
  virt_type = kvm

  Note
  ======
  The live-migration will work as expected if I change the nova.conf config to any one of the config below:
  1- Config1: 
  [libvirt]
  virt_type = qemu

  2- Config2:
  [libvirt]
  virt_type = kvm
  cpu_mode=custom
  cpu_models = Cascadelake-Server-noTSX,Skylake-Server-noTSX-IBRS,Broadwell-noTSX-IBRS

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1910331/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next