yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #96259
[Bug 2119542] [NEW] Nova fails to reshape compute node with cpu_dedicated_set after workload migration and zero allocations
Public bug reported:
Summary:
After configuring cpu_dedicated_set on a compute node in nova.conf, Nova fails to perform the required resource provider reshape, even when all instances have been migrated away, and Placement shows zero allocations. The error is persistent across service restarts and results in the compute node flapping between old and new inventory layouts (e.g., PCPU appears and disappears).
Steps to Reproduce:
1) Deploy OpenStack (tested on 2024.1 Caracal).
2) Schedule several instances to a compute node (e.g., compute01).
3) Migrate all instances off that node so it has no active workloads.
4) Verify openstack server list --all --host compute01 is empty.
5) Add the following to /etc/nova/nova.conf on compute01:
[compute]
cpu_shared_set = 0-17,36-53
cpu_dedicated_set = 18-35,54-71
6) Restart nova-compute service.
Expected Result:
Nova should detect the change, perform a reshape, and update Placement to report PCPU inventory.
Actual Result:
Nova logs show:
"2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task Traceback (most recent call last):
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task task(self, context)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10731, in update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_available_resource_for_node(context, nodename,
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10633, in _update_available_resource_for_node
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task with excutils.save_and_reraise_exception():
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self.force_reraise()
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task raise self.value
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10606, in _update_available_resource_for_node
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self.rt.update_available_resource(context, nodename,
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 935, in update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_available_resource(context, resources, startup=startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task return f(*args, **kwargs)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update(context, cn, startup=startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1375, in _update
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_to_placement(context, compute_node, startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task return Retrying(*dargs, **dkw).call(f, *args, **kw)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task return attempt.get(self._wrap_exception)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2])
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/six.py", line 719, in reraise
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task raise value
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1288, in _update_to_placement
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self.driver.update_provider_tree(prov_tree, nodename)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9284, in update_provider_tree
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_provider_tree_for_pcpu(
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9801, in _update_provider_tree_for_pcpu
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task raise exception.ReshapeNeeded()
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved.
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task
2025-08-04 19:15:52.787 7 ERROR nova.compute.manager [None req-caeacd66-7732-4826-89ea-f2231fed87aa - - - - - -] ReshapeNeeded exception is unexpected here!: nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved."
No reshape is performed.
Placement inventory flips between VCPU-only and VCPU+PCPU models.
No resource allocations exist on the host (confirmed via openstack
allocation list).
Additional Info:
The issue persists even after multiple restarts of nova-compute
Environment:
Nova version: 2024.1 (Caracal)
Placement version: 11.0.0
Deployment: Kolla-Ansible
Host: drained compute node with no allocations
CPU pinning being enabled for the first time (cpu_dedicated_set)
** Affects: nova
Importance: Undecided
Status: New
** Tags: nova
** Description changed:
Summary:
After configuring cpu_dedicated_set on a compute node in nova.conf, Nova fails to perform the required resource provider reshape, even when all instances have been migrated away, and Placement shows zero allocations. The error is persistent across service restarts and results in the compute node flapping between old and new inventory layouts (e.g., PCPU appears and disappears).
-
Steps to Reproduce:
1) Deploy OpenStack (tested on 2024.1 Caracal).
2) Schedule several instances to a compute node (e.g., compute01).
3) Migrate all instances off that node so it has no active workloads.
4) Verify openstack server list --all --host compute01 is empty.
5) Add the following to /etc/nova/nova.conf on compute01:
[compute]
- cpu_dedicated_set = 18-35
- cpu_shared_set = 0-17,36-71
+ cpu_shared_set = 0-17,36-53
+ cpu_dedicated_set = 18-35,54-71
+
6) Restart nova-compute service.
-
Expected Result:
Nova should detect the change, perform a reshape, and update Placement to report PCPU inventory.
Actual Result:
Nova logs show:
"2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task Traceback (most recent call last):
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task task(self, context)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10731, in update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_available_resource_for_node(context, nodename,
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10633, in _update_available_resource_for_node
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task with excutils.save_and_reraise_exception():
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self.force_reraise()
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task raise self.value
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10606, in _update_available_resource_for_node
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self.rt.update_available_resource(context, nodename,
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 935, in update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_available_resource(context, resources, startup=startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task return f(*args, **kwargs)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update(context, cn, startup=startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1375, in _update
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_to_placement(context, compute_node, startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task return Retrying(*dargs, **dkw).call(f, *args, **kw)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task return attempt.get(self._wrap_exception)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2])
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/six.py", line 719, in reraise
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task raise value
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1288, in _update_to_placement
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self.driver.update_provider_tree(prov_tree, nodename)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9284, in update_provider_tree
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_provider_tree_for_pcpu(
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9801, in _update_provider_tree_for_pcpu
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task raise exception.ReshapeNeeded()
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved.
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task
2025-08-04 19:15:52.787 7 ERROR nova.compute.manager [None req-caeacd66-7732-4826-89ea-f2231fed87aa - - - - - -] ReshapeNeeded exception is unexpected here!: nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved."
-
No reshape is performed.
Placement inventory flips between VCPU-only and VCPU+PCPU models.
No resource allocations exist on the host (confirmed via openstack
allocation list).
-
Additional Info:
The issue persists even after multiple restarts of nova-compute
-
Environment:
Nova version: 2024.1 (Caracal)
Placement version: 11.0.0
Deployment: Kolla-Ansible
Host: drained compute node with no allocations
CPU pinning being enabled for the first time (cpu_dedicated_set)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2119542
Title:
Nova fails to reshape compute node with cpu_dedicated_set after
workload migration and zero allocations
Status in OpenStack Compute (nova):
New
Bug description:
Summary:
After configuring cpu_dedicated_set on a compute node in nova.conf, Nova fails to perform the required resource provider reshape, even when all instances have been migrated away, and Placement shows zero allocations. The error is persistent across service restarts and results in the compute node flapping between old and new inventory layouts (e.g., PCPU appears and disappears).
Steps to Reproduce:
1) Deploy OpenStack (tested on 2024.1 Caracal).
2) Schedule several instances to a compute node (e.g., compute01).
3) Migrate all instances off that node so it has no active workloads.
4) Verify openstack server list --all --host compute01 is empty.
5) Add the following to /etc/nova/nova.conf on compute01:
[compute]
cpu_shared_set = 0-17,36-53
cpu_dedicated_set = 18-35,54-71
6) Restart nova-compute service.
Expected Result:
Nova should detect the change, perform a reshape, and update Placement to report PCPU inventory.
Actual Result:
Nova logs show:
"2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task Traceback (most recent call last):
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_service/periodic_task.py", line 216, in run_periodic_tasks
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task task(self, context)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10731, in update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_available_resource_for_node(context, nodename,
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10633, in _update_available_resource_for_node
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task with excutils.save_and_reraise_exception():
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self.force_reraise()
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task raise self.value
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/manager.py", line 10606, in _update_available_resource_for_node
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self.rt.update_available_resource(context, nodename,
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 935, in update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_available_resource(context, resources, startup=startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/oslo_concurrency/lockutils.py", line 412, in inner
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task return f(*args, **kwargs)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1066, in _update_available_resource
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update(context, cn, startup=startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1375, in _update
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_to_placement(context, compute_node, startup)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 56, in wrapped_f
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task return Retrying(*dargs, **dkw).call(f, *args, **kw)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 257, in call
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task return attempt.get(self._wrap_exception)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 301, in get
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task six.reraise(self.value[0], self.value[1], self.value[2])
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/six.py", line 719, in reraise
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task raise value
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/retrying.py", line 251, in call
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/compute/resource_tracker.py", line 1288, in _update_to_placement
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self.driver.update_provider_tree(prov_tree, nodename)
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9284, in update_provider_tree
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task self._update_provider_tree_for_pcpu(
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task File "/var/lib/kolla/venv/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 9801, in _update_provider_tree_for_pcpu
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task raise exception.ReshapeNeeded()
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved.
2025-08-04 19:14:51.723 7 ERROR oslo_service.periodic_task
2025-08-04 19:15:52.787 7 ERROR nova.compute.manager [None req-caeacd66-7732-4826-89ea-f2231fed87aa - - - - - -] ReshapeNeeded exception is unexpected here!: nova.exception.ReshapeNeeded: Virt driver indicates that provider inventories need to be moved."
No reshape is performed.
Placement inventory flips between VCPU-only and VCPU+PCPU models.
No resource allocations exist on the host (confirmed via openstack
allocation list).
Additional Info:
The issue persists even after multiple restarts of nova-compute
Environment:
Nova version: 2024.1 (Caracal)
Placement version: 11.0.0
Deployment: Kolla-Ansible
Host: drained compute node with no allocations
CPU pinning being enabled for the first time (cpu_dedicated_set)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2119542/+subscriptions