yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2009280] Re: evmc not supported on AMD processors but is enabled by default for "windows" instances

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <2009280@xxxxxxxxxxxxxxxxxx>
Date: Thu, 15 Feb 2024 10:22:48 -0000
Reply-to: Bug 2009280 <2009280@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/nova/+/899776
Committed: https://opendev.org/openstack/nova/commit/86a35e97d286cbb6e23f8cc7bec5a05f022da0cb
Submitter: "Zuul (22348)"
Branch:    master

commit 86a35e97d286cbb6e23f8cc7bec5a05f022da0cb
Author: Artom Lifshitz <alifshit@xxxxxxxxxx>
Date:   Tue Oct 31 22:52:50 2023 -0400

    libvirt: Stop unconditionally enabling evmcs
    
    In I008841988547573878c4e06e82f0fa55084e51b5 we started enabling a
    bunch of libvirt enlightenments for Windows unconditionally. Turns
    out, the `evmcs` enlightenment only works on Intel hosts, and we broke
    the ability to run Windows guests on AMD machines. Until we become
    smarter about conditionally enabling evmcs (with something like traits
    for host CPU features), just stop enabling it at all.
    
    Change-Id: I2ff4fdecd9dc69de283f0e52e07df1aeaf0a9048
    Closes-bug: 2009280


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2009280

Title:
  evmc not supported on AMD processors but is enabled by default for
  "windows" instances

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  on AMD Processors it is known that QEMU does not not support hv-evmc
  but as of notes/update-libvirt-enlightenments-for-windows-23abea98cc1db667.yaml it was enabled

  There is no way that I can see to single out this flag and nova should
  check if the remote CPU is AMD and disable this enablement.

  You can test this on any AMD machine with "qemu-system-x86_64
  --nographic -cpu
  host,hv_relaxed,hv_vapic,hv_spinlocks=8191,hv_vpindex,hv_runtime,hv_synic,hv_stimer,hv_reset,hv_frequencies,hv_time,hv_evmcs
  --enable-kvm" and it will throw a critical error of "Hyper-V
  enlightened VMCS (hv-evmcs) is not supported by kernel"

  
  Here are example logs that are thrown by nova when trying to start the instance, the instance was set to os_type: windows to have it throw this. This was tested on Zed with a AMD EYPC 7702

  
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] [instance: d416180c-cd56-4830-9787-3bb4953d9b02] Failed to build and run instance: libvirt.libvirt]
  2023-03-04T20:57:46.694416Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.tsc-scale [bit 4]
  2023-03-04T20:57:46.694441Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vmcb-clean [bit 5]
  2023-03-04T20:57:46.694465Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pause-filter [bit 10]
  2023-03-04T20:57:46.694489Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pfthreshold [bit 12]
  2023-03-04T20:57:46.694513Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
  2023-03-04T20:57:46.694549Z qemu-kvm: Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] Traceback (most recent call last):
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/compute/manager.py", line 2517, in _build_and_run_instance
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     self.driver.spawn(context, instance, image_meta,
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4366, in spawn
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     self._create_guest_with_network(
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7724, in _create_guest_with_network
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     self._cleanup(
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     self.force_reraise()
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     raise self.value
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7701, in _create_guest_with_network
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     guest = self._create_guest(
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7640, in _create_guest
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     guest.launch(pause=pause)
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 168, in launch
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     LOG.exception('Error launching a defined domain with XML: %s',
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     self.force_reraise()
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     raise self.value
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 165, in launch
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     return self._domain.createWithFlags(flags)
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 193, in doit
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 151, in proxy_call
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     rv = execute(f, *args, **kwargs)
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     six.reraise(c, e, tb)
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     raise value
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     rv = meth(*args, **kwargs)
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]   File "/usr/lib64/python3.9/site-packages/libvirt.py", line 1398, in createWithFlags
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]     raise libvirtError('virDomainCreateWithFlags() failed')
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2023-03-04T20:57:46.694375Z qemu-kvm: warning: This feature depends on other features that were not request]
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694416Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.tsc-scale [bit 4]
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694441Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.vmcb-clean [bit 5]
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694465Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pause-filter [bit 10]
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694489Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.pfthreshold [bit 12]
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694513Z qemu-kvm: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02] 2023-03-04T20:57:46.694549Z qemu-kvm: Hyper-V enlightened VMCS (hv-evmcs) is not supported by kernel
  2023-03-04 15:57:47.721 7 ERROR nova.compute.manager [instance: d416180c-cd56-4830-9787-3bb4953d9b02]
  2023-03-04 15:57:47.726 7 INFO os_vif [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] Successfully unplugged vif VIFOpenVSwitch(active=False,address=fa:16:3e:23:e5:4f,bridge_name='br-int',has_traffic)
  2023-03-04 15:57:47.981 7 INFO nova.compute.manager [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] [instance: d416180c-cd56-4830-9787-3bb4953d9b02] Took 0.26 seconds to deallocate network for instan.
  2023-03-04 15:57:48.060 7 INFO nova.scheduler.client.report [None req-2172765f-0769-4b82-88fa-3cd68144ee42 ff259b03efc14bee83c881bc4920a12d a69aa169729a4dbea0d9b9683072260d - - default default] Deleted allocations for instance d416180c-cd56-4830-9787-3bb4953d9b02
  2023-03-04 15:57:48.079 7 INFO nova.compute.manager [-] [instance: d416180c-cd56-4830-9787-3bb4953d9b02] During sync_power_state the instance has a pending task (block_device_mapping). Skip.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2009280/+subscriptions
References

[Bug 2009280] [NEW] evmc not supported on AMD processors but is enabled by default for "windows" instances
From: Forrest Fuqua, 2023-03-04