yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2116882] [NEW] nova compute doesn't respect cpu_model_extra_flags

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lana Kaleif <2116882@xxxxxxxxxxxxxxxxxx>
Date: Mon, 14 Jul 2025 15:14:58 -0000
Reply-to: Bug 2116882 <2116882@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

nova compute doesn't respect cpu_model_extra_flags while comparing CPU feature flag sets of spawned VMs  with HV
(https://github.com/openstack/nova/blob/ad1bd175ae57e45dbfa8014f945cbc42d62ad8ce/nova/virt/libvirt/driver.py#L5714)

Environment:
OS: Almalinux 9.5
Openstack version: Caracal
openstack-nova-compute version 29.2.0

nova compute configuration:

```
cpu_mode = custom
cpu_models = Cascadelake-Server-noTSX
cpu_model_extra_flags = vmx,pcid
```

NOTE: Cascadelake-Server-noTSX custom model does not provide VMX flag
(needed for nested virtualization) and is compatible with used compute
node CPU

Test scenario 1 (cpu_model_extra_flags is not used):

1. Remove cpu_model_extra_flags from nova config, restart nova compute service
2. VMX flag does not appear in placement resource list
```
openstack resource provider trait list 1df5827d-c8cb-4239-bf91-49752635a2a8 | grep VMX
| COMPUTE_NET_VIF_MODEL_VMXNET3         |
```

3. Create a new flavor, or modify existing one with VMX trait requirement in metadata:
```
openstack flavor set m1.large --property "trait:HW_CPU_X86_VMX"="required"
```
4. Create a server with a flavor from step 3
```
openstack server create --flavor m1.large --image 46357e8f-b90f-401f-9df1-8eaa3922ba7 --network private_network --os-compute-api-version 2.74 --key-name DEPLOYER_key --security-group 13b61e70-d2fd-4c63-81cb-c67d282e8b2a vmxtest
```
5. VM creation fails on scheduling stage

This behavior is expected since no HVs are reporting a compatible list
of CPU flags for this VM

Test scenario 2 (cpu_model_extra_flags is set to `vmx,pcid`)

1. Set cpu_model_extra_flags to `vmx,pcid`, restart nova compute service
2. VMX flag appears in placement resource list
```
openstack resource provider trait list 1df5827d-c8cb-4239-bf91-49752635a2a8 | grep VMX
| HW_CPU_X86_VMX                        |
| HW_CPU_X86_INTEL_VMX                  |
```
3. Create a new flavor, or modify existing one with VMX trait requirement in metadata:
```
openstack flavor set m1.large --property "trait:HW_CPU_X86_VMX"="required"
```
4. Create a server with a flavor from step 3
```
openstack server create --flavor m1.large --image 46357e8f-b90f-401f-9df1-8eaa3922ba7 --network private_network --os-compute-api-version 2.74 --key-name DEPLOYER_key --security-group 13b61e70-d2fd-4c63-81cb-c67d282e8b2a vmxtest
```
5. VM is scheduled to correct HV (one with cpu_model_extra_flags set to vmx,pcid)
6. VM spawn is failed on compute host due to CPU flags incompatibility:

```
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [None req-d9edc3cc-18f5-4907-b491-fcaafcec1e9e bf902e2d47e241fa8c812a8844c21cdf 16f06b95007248c98d6b1dec1581088b - - default default] [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] Instance failed to spawn: nova.exception.InvalidCPUInfo: No CPU model match traits, models: ['Cascadelake-Server-noTSX'], required flags: {'vmx'}
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] Traceback (most recent call last):
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2885, in _build_resources
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     yield resources
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2632, in _build_and_run_instance
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     self.driver.spawn(context, instance, image_meta,
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4638, in spawn
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     xml = self._get_guest_xml(context, instance, network_info,
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7817, in _get_guest_xml
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     conf = self._get_guest_config(instance, network_info, image_meta,
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7297, in _get_guest_config
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     guest.cpu = self._get_guest_cpu_config(
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5709, in _get_guest_cpu_config
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     cpu = self._get_guest_cpu_model_config(flavor, arch)
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5634, in _get_guest_cpu_model_config
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     cpu.model = self._match_cpu_model_by_flags(models, flags)
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5704, in _match_cpu_model_by_flags
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     raise exception.InvalidCPUInfo(msg)
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] nova.exception.InvalidCPUInfo: No CPU model match traits, models: ['Cascadelake-Server-noTSX'], required flags: {'vmx'}
2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] 
2025-06-25 13:00:48.129 2493382 INFO nova.compute.manager [None req-d9edc3cc-18f5-4907-b491-fcaafcec1e9e bf902e2d47e241fa8c812a8844c21cdf 16f06b95007248c98d6b1dec1581088b - - default default] [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] Terminating instance
```

Expected behavior:
The VM is spawned without errors on the HV with cpu_model_extra_flags is set to `vmx,pcid`

This happens in https://github.com/openstack/nova/blob/ad1bd175ae57e45dbfa8014f945cbc42d62ad8ce/nova/virt/libvirt/driver.py#L5714
namely because of requested `flags` set is not a subset `feature_names` which is a flag set of a CPU model configured in nova.conf

According to nova docs cpu_model_extra_flags should allow adding flags to pre-defined custom models, which it seemingly does not do
(https://docs.openstack.org/nova/latest/admin/cpu-models.html#cpu-feature-flags)

Proposed patch is attached. It simply adds a flag set from
cpu_model_extra_flags to the list derived from baseline CPU and then
used for comparison, which seems consistent with documented logic

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: nova nova-compute

** Patch added: "nova_extra_flags.patch"
   https://bugs.launchpad.net/bugs/2116882/+attachment/5889712/+files/nova_extra_flags.patch

** Tags added: nova

** Tags added: nova-compute

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2116882

Title:
  nova compute doesn't respect cpu_model_extra_flags

Status in OpenStack Compute (nova):
  New

Bug description:
  nova compute doesn't respect cpu_model_extra_flags while comparing CPU feature flag sets of spawned VMs  with HV
  (https://github.com/openstack/nova/blob/ad1bd175ae57e45dbfa8014f945cbc42d62ad8ce/nova/virt/libvirt/driver.py#L5714)

  Environment:
  OS: Almalinux 9.5
  Openstack version: Caracal
  openstack-nova-compute version 29.2.0

  nova compute configuration:

  ```
  cpu_mode = custom
  cpu_models = Cascadelake-Server-noTSX
  cpu_model_extra_flags = vmx,pcid
  ```

  NOTE: Cascadelake-Server-noTSX custom model does not provide VMX flag
  (needed for nested virtualization) and is compatible with used compute
  node CPU

  Test scenario 1 (cpu_model_extra_flags is not used):

  1. Remove cpu_model_extra_flags from nova config, restart nova compute service
  2. VMX flag does not appear in placement resource list
  ```
  openstack resource provider trait list 1df5827d-c8cb-4239-bf91-49752635a2a8 | grep VMX
  | COMPUTE_NET_VIF_MODEL_VMXNET3         |
  ```

  3. Create a new flavor, or modify existing one with VMX trait requirement in metadata:
  ```
  openstack flavor set m1.large --property "trait:HW_CPU_X86_VMX"="required"
  ```
  4. Create a server with a flavor from step 3
  ```
  openstack server create --flavor m1.large --image 46357e8f-b90f-401f-9df1-8eaa3922ba7 --network private_network --os-compute-api-version 2.74 --key-name DEPLOYER_key --security-group 13b61e70-d2fd-4c63-81cb-c67d282e8b2a vmxtest
  ```
  5. VM creation fails on scheduling stage

  This behavior is expected since no HVs are reporting a compatible list
  of CPU flags for this VM

  Test scenario 2 (cpu_model_extra_flags is set to `vmx,pcid`)

  1. Set cpu_model_extra_flags to `vmx,pcid`, restart nova compute service
  2. VMX flag appears in placement resource list
  ```
  openstack resource provider trait list 1df5827d-c8cb-4239-bf91-49752635a2a8 | grep VMX
  | HW_CPU_X86_VMX                        |
  | HW_CPU_X86_INTEL_VMX                  |
  ```
  3. Create a new flavor, or modify existing one with VMX trait requirement in metadata:
  ```
  openstack flavor set m1.large --property "trait:HW_CPU_X86_VMX"="required"
  ```
  4. Create a server with a flavor from step 3
  ```
  openstack server create --flavor m1.large --image 46357e8f-b90f-401f-9df1-8eaa3922ba7 --network private_network --os-compute-api-version 2.74 --key-name DEPLOYER_key --security-group 13b61e70-d2fd-4c63-81cb-c67d282e8b2a vmxtest
  ```
  5. VM is scheduled to correct HV (one with cpu_model_extra_flags set to vmx,pcid)
  6. VM spawn is failed on compute host due to CPU flags incompatibility:

  ```
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [None req-d9edc3cc-18f5-4907-b491-fcaafcec1e9e bf902e2d47e241fa8c812a8844c21cdf 16f06b95007248c98d6b1dec1581088b - - default default] [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] Instance failed to spawn: nova.exception.InvalidCPUInfo: No CPU model match traits, models: ['Cascadelake-Server-noTSX'], required flags: {'vmx'}
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] Traceback (most recent call last):
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2885, in _build_resources
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     yield resources
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 2632, in _build_and_run_instance
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     self.driver.spawn(context, instance, image_meta,
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 4638, in spawn
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     xml = self._get_guest_xml(context, instance, network_info,
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7817, in _get_guest_xml
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     conf = self._get_guest_config(instance, network_info, image_meta,
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 7297, in _get_guest_config
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     guest.cpu = self._get_guest_cpu_config(
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5709, in _get_guest_cpu_config
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     cpu = self._get_guest_cpu_model_config(flavor, arch)
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5634, in _get_guest_cpu_model_config
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     cpu.model = self._match_cpu_model_by_flags(models, flags)
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 5704, in _match_cpu_model_by_flags
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979]     raise exception.InvalidCPUInfo(msg)
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] nova.exception.InvalidCPUInfo: No CPU model match traits, models: ['Cascadelake-Server-noTSX'], required flags: {'vmx'}
  2025-06-25 13:00:48.126 2493382 ERROR nova.compute.manager [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] 
  2025-06-25 13:00:48.129 2493382 INFO nova.compute.manager [None req-d9edc3cc-18f5-4907-b491-fcaafcec1e9e bf902e2d47e241fa8c812a8844c21cdf 16f06b95007248c98d6b1dec1581088b - - default default] [instance: d747a173-2bb9-4794-8ea2-c5ab298b4979] Terminating instance
  ```

  Expected behavior:
  The VM is spawned without errors on the HV with cpu_model_extra_flags is set to `vmx,pcid`

  This happens in https://github.com/openstack/nova/blob/ad1bd175ae57e45dbfa8014f945cbc42d62ad8ce/nova/virt/libvirt/driver.py#L5714
  namely because of requested `flags` set is not a subset `feature_names` which is a flag set of a CPU model configured in nova.conf

  According to nova docs cpu_model_extra_flags should allow adding flags to pre-defined custom models, which it seemingly does not do
  (https://docs.openstack.org/nova/latest/admin/cpu-models.html#cpu-feature-flags)

  Proposed patch is attached. It simply adds a flag set from
  cpu_model_extra_flags to the list derived from baseline CPU and then
  used for comparison, which seems consistent with documented logic

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2116882/+subscriptions