yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #92468
[Bug 2023035] [NEW] 2023.1: Live migration fails with new compare_hypervisor_cpu method
Public bug reported:
We have test setup with 3 identical compute nodes. Recently test setup
upgraded from Zed to 2023.1 Openstack Version, with nova 27.0.0, libvirt
8.0.0, qemu 6.2
During live migration we always getting error:
2023-06-05 13:33:06.158 1458525 ERROR nova.virt.libvirt.driver [None
req-6295f150-f0cf-41d1-8dd4-60c0ac32223f
59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - -
b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] CPU
doesn't have compatibility.
0
Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server [None req-6295f150-f0cf-41d1-8dd4-60c0ac32223f 59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - - b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] Exception during message handling: nova.exception.MigrationPreCheckEr
ror: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.
0
Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9637, in check_can_live_migrate_destination
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self._compare_cpu(None, source_cpu_info, instance)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10014, in _compare_cpu
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u})
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 0
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 65, in wrapped
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise self.value
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 213, in decorated_function
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise self.value
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 203, in decorated_function
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 8408, in check_can_live_migrate_destination
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9641, in check_can_live_migrate_destination
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise exception.MigrationPreCheckError(reason=e)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.
Zed setup works ok with this nodes.
More info:
1. https://github.com/openstack/nova/commit/468b03e0ee4a917ae26106f6e57081bcd9e7a65b
If revert libvirt to old cpu_compare instead cpu_hypervisor_compare in libvirt driver
ret = self._host.compare_cpu(cpu_xml)
#ret = self._host.compare_hypervisor_cpu(cpu_xml)
live migration works ok.
ret = self._host.compare_cpu(cpu_xml) - always returns 2 (VIR_CPU_COMPARE_SUPERSET)
ret = self._host.compare_hypervisor_cpu(cpu_xml) - always returns 0 (VIR_CPU_COMPARE_INCOMPATIBLE)
2. Disabling cpu Comparation with changing nova.conf:
[workarounds]
skip_cpu_compare_on_dest = True
makes live migration work ok.
3. Entry xml to cpu_compare looks like:
<cpu>
<arch>x86_64</arch>
<model>Skylake-Client-noTSX-IBRS</model>
<vendor>Intel</vendor>
<topology sockets="1" cores="4" threads="2"/>
<feature name="3dnowprefetch"/>
<feature name="abm"/>
<feature name="acpi"/>
<feature name="adx"/>
<feature name="aes"/>
<feature name="apic"/>
<feature name="arat"/>
<feature name="arch-capabilities"/>
<feature name="avx"/>
<feature name="avx2"/>
<feature name="bmi1"/>
<feature name="bmi2"/>
<feature name="clflush"/>
<feature name="clflushopt"/>
<feature name="cmov"/>
<feature name="cx16"/>
<feature name="cx8"/>
<feature name="de"/>
<feature name="ds"/>
<feature name="ds_cpl"/>
<feature name="dtes64"/>
<feature name="erms"/>
<feature name="est"/>
<feature name="f16c"/>
<feature name="fma"/>
<feature name="fpu"/>
<feature name="fsgsbase"/>
<feature name="fxsr"/>
<feature name="ht"/>
<feature name="intel-pt"/>
<feature name="invpcid"/>
<feature name="invtsc"/>
<feature name="lahf_lm"/>
<feature name="lm"/>
<feature name="mca"/>
<feature name="mce"/>
<feature name="md-clear"/>
<feature name="mmx"/>
<feature name="monitor"/>
<feature name="movbe"/>
<feature name="mpx"/>
<feature name="msr"/>
<feature name="mtrr"/>
<feature name="nx"/>
<feature name="pae"/>
<feature name="pat"/>
<feature name="pbe"/>
<feature name="pcid"/>
<feature name="pclmuldq"/>
<feature name="pdcm"/>
<feature name="pdpe1gb"/>
<feature name="pge"/>
<feature name="pni"/>
<feature name="popcnt"/>
<feature name="pse"/>
<feature name="pse36"/>
<feature name="rdrand"/>
<feature name="rdseed"/>
<feature name="rdtscp"/>
<feature name="rsba"/>
<feature name="sep"/>
<feature name="smap"/>
<feature name="smep"/>
<feature name="smx"/>
<feature name="spec-ctrl"/>
<feature name="ss"/>
<feature name="ssbd"/>
<feature name="sse"/>
<feature name="sse2"/>
<feature name="sse4.1"/>
<feature name="sse4.2"/>
<feature name="ssse3"/>
<feature name="stibp"/>
<feature name="syscall"/>
<feature name="tm"/>
<feature name="tm2"/>
<feature name="tsc"/>
<feature name="tsc-deadline"/>
<feature name="tsc_adjust"/>
<feature name="vme"/>
<feature name="vmx"/>
<feature name="x2apic"/>
<feature name="xgetbv1"/>
<feature name="xsave"/>
<feature name="xsavec"/>
<feature name="xsaveopt"/>
<feature name="xsaves"/>
<feature name="xtpr"/>
</cpu>
nova.conf
[libvirt]
cpu_mode = host-model
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2023035
Title:
2023.1: Live migration fails with new compare_hypervisor_cpu method
Status in OpenStack Compute (nova):
New
Bug description:
We have test setup with 3 identical compute nodes. Recently test setup
upgraded from Zed to 2023.1 Openstack Version, with nova 27.0.0,
libvirt 8.0.0, qemu 6.2
During live migration we always getting error:
2023-06-05 13:33:06.158 1458525 ERROR nova.virt.libvirt.driver [None
req-6295f150-f0cf-41d1-8dd4-60c0ac32223f
59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - -
b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] CPU
doesn't have compatibility.
0
Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server [None req-6295f150-f0cf-41d1-8dd4-60c0ac32223f 59de9e2e2a8a413384be5ee27e027fc1 185764021e19409dae135a967f032fa4 - - b589410bd7e14872bf3ac74c45057691 b589410bd7e14872bf3ac74c45057691] Exception during message handling: nova.exception.MigrationPreCheckEr
ror: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.
0
Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9637, in check_can_live_migrate_destination
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self._compare_cpu(None, source_cpu_info, instance)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 10014, in _compare_cpu
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise exception.InvalidCPUInfo(reason=m % {'ret': ret, 'u': u})
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.InvalidCPUInfo: Unacceptable CPU info: CPU doesn't have compatibility.
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server 0
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Refer to http://libvirt.org/html/libvirt-libvirt-host.html#virCPUCompareResult
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 65, in wrapped
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise self.value
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/exception_wrapper.py", line 63, in wrapped
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1439, in decorated_function
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 213, in decorated_function
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server self.force_reraise()
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise self.value
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 203, in decorated_function
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/manager.py", line 8408, in check_can_live_migrate_destination
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server dest_check_data = self.driver.check_can_live_migrate_destination(ctxt,
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 9641, in check_can_live_migrate_destination
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server raise exception.MigrationPreCheckError(reason=e)
2023-06-05 13:33:06.264 1458525 ERROR oslo_messaging.rpc.server nova.exception.MigrationPreCheckError: Migration pre-check error: Unacceptable CPU info: CPU doesn't have compatibility.
Zed setup works ok with this nodes.
More info:
1. https://github.com/openstack/nova/commit/468b03e0ee4a917ae26106f6e57081bcd9e7a65b
If revert libvirt to old cpu_compare instead cpu_hypervisor_compare in libvirt driver
ret = self._host.compare_cpu(cpu_xml)
#ret = self._host.compare_hypervisor_cpu(cpu_xml)
live migration works ok.
ret = self._host.compare_cpu(cpu_xml) - always returns 2 (VIR_CPU_COMPARE_SUPERSET)
ret = self._host.compare_hypervisor_cpu(cpu_xml) - always returns 0 (VIR_CPU_COMPARE_INCOMPATIBLE)
2. Disabling cpu Comparation with changing nova.conf:
[workarounds]
skip_cpu_compare_on_dest = True
makes live migration work ok.
3. Entry xml to cpu_compare looks like:
<cpu>
<arch>x86_64</arch>
<model>Skylake-Client-noTSX-IBRS</model>
<vendor>Intel</vendor>
<topology sockets="1" cores="4" threads="2"/>
<feature name="3dnowprefetch"/>
<feature name="abm"/>
<feature name="acpi"/>
<feature name="adx"/>
<feature name="aes"/>
<feature name="apic"/>
<feature name="arat"/>
<feature name="arch-capabilities"/>
<feature name="avx"/>
<feature name="avx2"/>
<feature name="bmi1"/>
<feature name="bmi2"/>
<feature name="clflush"/>
<feature name="clflushopt"/>
<feature name="cmov"/>
<feature name="cx16"/>
<feature name="cx8"/>
<feature name="de"/>
<feature name="ds"/>
<feature name="ds_cpl"/>
<feature name="dtes64"/>
<feature name="erms"/>
<feature name="est"/>
<feature name="f16c"/>
<feature name="fma"/>
<feature name="fpu"/>
<feature name="fsgsbase"/>
<feature name="fxsr"/>
<feature name="ht"/>
<feature name="intel-pt"/>
<feature name="invpcid"/>
<feature name="invtsc"/>
<feature name="lahf_lm"/>
<feature name="lm"/>
<feature name="mca"/>
<feature name="mce"/>
<feature name="md-clear"/>
<feature name="mmx"/>
<feature name="monitor"/>
<feature name="movbe"/>
<feature name="mpx"/>
<feature name="msr"/>
<feature name="mtrr"/>
<feature name="nx"/>
<feature name="pae"/>
<feature name="pat"/>
<feature name="pbe"/>
<feature name="pcid"/>
<feature name="pclmuldq"/>
<feature name="pdcm"/>
<feature name="pdpe1gb"/>
<feature name="pge"/>
<feature name="pni"/>
<feature name="popcnt"/>
<feature name="pse"/>
<feature name="pse36"/>
<feature name="rdrand"/>
<feature name="rdseed"/>
<feature name="rdtscp"/>
<feature name="rsba"/>
<feature name="sep"/>
<feature name="smap"/>
<feature name="smep"/>
<feature name="smx"/>
<feature name="spec-ctrl"/>
<feature name="ss"/>
<feature name="ssbd"/>
<feature name="sse"/>
<feature name="sse2"/>
<feature name="sse4.1"/>
<feature name="sse4.2"/>
<feature name="ssse3"/>
<feature name="stibp"/>
<feature name="syscall"/>
<feature name="tm"/>
<feature name="tm2"/>
<feature name="tsc"/>
<feature name="tsc-deadline"/>
<feature name="tsc_adjust"/>
<feature name="vme"/>
<feature name="vmx"/>
<feature name="x2apic"/>
<feature name="xgetbv1"/>
<feature name="xsave"/>
<feature name="xsavec"/>
<feature name="xsaveopt"/>
<feature name="xsaves"/>
<feature name="xtpr"/>
</cpu>
nova.conf
[libvirt]
cpu_mode = host-model
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2023035/+subscriptions