yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #95775
[Bug 2100342] Re: Unable to overcommit CPU
Just making this one as invalid as it looks like a duplicate of #2100343
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2100342
Title:
Unable to overcommit CPU
Status in OpenStack Compute (nova):
Invalid
Bug description:
Cannot overcommit CPU - cpu_allocation_ratio in nova.conf is ignored.
I am using Dalmatian release on Ubuntu 24.04
Steps to reproduce:
1. cpu_allocation_ratio = 16 in /etc/nova/nova.conf
2. restart nova-schedular and nova-compute
3. try provision vm so that the total vcpu count exceeds the physical cpu count in host.
Following is from nova-conductor.log
---------
2025-02-27 02:20:54.419 235713 WARNING nova.scheduler.utils [None req-3518a8f4-f937-4475-a80d-c05a89278c35 91df74a173864580bc23dc8e5d95c602 8608366cdbc7478c98796c3c416a976b - - default default] Failed to compute_task_build_instances: Quota exceeded for cores: Requested 24, but already used 100 of 100 cores: nova.exception.TooManyInstances: Quota exceeded for cores: Requested 24, but already used 100 of 100 cores
2025-02-27 02:20:54.423 235713 WARNING nova.scheduler.utils [None req-3518a8f4-f937-4475-a80d-c05a89278c35 91df74a173864580bc23dc8e5d95c602 8608366cdbc7478c98796c3c416a976b - - default default] [instance: 245e25e2-6c70-45b5-992f-11dbb240c854] Setting instance to ERROR state.: nova.exception.TooManyInstances: Quota exceeded for cores: Requested 24, but already used 100 of 100 cores
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server [None req-3518a8f4-f937-4475-a80d-c05a89278c35 91df74a173864580bc23dc8e5d95c602 8608366cdbc7478c98796c3c416a976b - - default default] Exception during message handling: nova.exception.TooManyInstances: Quota exceeded for cores: Requested 24, but already used 100 of 100 cores
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1129, in check_num_instances_quota
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server objects.Quotas.check_deltas(context, deltas,
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server result = fn(cls, context, *args, **kwargs)
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/objects/quotas.py", line 370, in check_deltas
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server raise exc
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/objects/quotas.py", line 365, in check_deltas
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server cls.limit_check_project_and_user(context, **check_kwargs)
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server result = fn(cls, context, *args, **kwargs)
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/objects/quotas.py", line 298, in limit_check_project_and_user
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server return quota.QUOTAS.limit_check_project_and_user(context,
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/quota.py", line 1150, in limit_check_project_and_user
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server return self._driver.limit_check_project_and_user(
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/quota.py", line 588, in limit_check_project_and_user
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server raise exception.OverQuota(overs=sorted(overs),
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server nova.exception.OverQuota: Quota exceeded for resources: ['cores']
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 172, in _process_incoming
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server ^^^^^^^^^^^^^^^^^^^^^^
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1745, in schedule_and_build_instances
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server with excutils.save_and_reraise_exception():
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server self.force_reraise()
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server raise self.value
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 1739, in schedule_and_build_instances
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server self._recheck_quota(context, instance.flavor, request_specs[0],
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 996, in _recheck_quota
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server compute_utils.check_num_instances_quota(
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/nova/compute/utils.py", line 1151, in check_num_instances_quota
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server raise exception.TooManyInstances(overs=overs,
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server nova.exception.TooManyInstances: Quota exceeded for cores: Requested 24, but already used 100 of 100 cores
2025-02-27 02:20:54.493 235713 ERROR oslo_messaging.rpc.server
-------
I am trying to provision vm with 24 vCPU - 24+28 exceeds 48 CPU count. I can provision vm with 16 vCPU
Environment Details
------
root@cnsdenlabos01:/etc/nova# openstack resource provider inventory list 2f62a748-69b8-41cd-adda-5c9bafebff6a
+----------------+------------------+----------+----------+----------+-----------+--------+-------+
| resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used |
+----------------+------------------+----------+----------+----------+-----------+--------+-------+
| VCPU | 16.0 | 1 | 48 | 0 | 1 | 48 | 28 |
| MEMORY_MB | 1.5 | 1 | 773922 | 512 | 1 | 773922 | 53248 |
| DISK_GB | 1.0 | 1 | 1133 | 0 | 1 | 1133 | 170 |
+----------------+------------------+----------+----------+----------+-----------+--------+-------+
root@cnsdenlabos01:/etc/nova# openstack hypervisor list
+--------------------------------------+---------------------+-----------------+-------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+--------------------------------------+---------------------+-----------------+-------------+-------+
| 2f62a748-69b8-41cd-adda-5c9bafebff6a | cnsdenlabos01 | QEMU | 172.16.0.31 | up |
| 2f62a748-69b8-41cd-adda-5c9bafebff6a | cnsdenlabos01 | QEMU | 172.16.0.31 | up |
+--------------------------------------+---------------------+-----------------+-------------+-------+
root@cnsdenlabos01:/etc/nova# openstack hypervisor show 2f62a748-69b8-41cd-adda-5c9bafebff6a
+---------------------+--------------------------------------+
| Field | Value |
+---------------------+--------------------------------------+
| aggregates | [] |
| cpu_info | None |
| host_ip | 172.16.0.31 |
| host_time | 02:06:44 |
| hypervisor_hostname | cnsdenlabos01 |
| hypervisor_type | QEMU |
| hypervisor_version | 8002002 |
| id | 2f62a748-69b8-41cd-adda-5c9bafebff6a |
| load_average | 2.72, 2.84, 2.74 |
| service_host | cnsdenlabos01 |
| service_id | 088b7464-3101-4db7-92b8-2eac6e01fbf6 |
| state | up |
| status | enabled |
| uptime | 21:21 |
| users | 7 |
+---------------------+--------------------------------------+
root@cnsdenlabos01:/etc/nova# openstack host show cnsdenlabos01
API has been deprecated; consider using 'hypervisor show' instead.
+---------------+----------------------------------+-----+-----------+---------+
| Host | Project | CPU | Memory MB | Disk GB |
+---------------+----------------------------------+-----+-----------+---------+
| cnsdenlabos01 | (total) | 48 | 773922 | 1133 |
| cnsdenlabos01 | (used_now) | 28 | 53760 | 170 |
| cnsdenlabos01 | (used_max) | 28 | 53248 | 385 |
| cnsdenlabos01 | 8608366cdbc7478c98796c3c416a976b | 28 | 53248 | 385 |
+---------------+----------------------------------+-----+-----------+---------+
root@cnsdenlabos01:/etc/nova# dpkg -l | grep nova
ii nova-api 3:30.0.0-0ubuntu1~cloud0 all OpenStack Compute - API frontend
ii nova-common 3:30.0.0-0ubuntu1~cloud0 all OpenStack Compute - common files
ii nova-compute 3:30.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node base
ii nova-compute-kvm 3:30.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 3:30.0.0-0ubuntu1~cloud0 all OpenStack Compute - compute node libvirt support
ii nova-conductor 3:30.0.0-0ubuntu1~cloud0 all OpenStack Compute - conductor service
ii nova-novncproxy 3:30.0.0-0ubuntu1~cloud0 all OpenStack Compute - NoVNC proxy
ii nova-scheduler 3:30.0.0-0ubuntu1~cloud0 all OpenStack Compute - virtual machine scheduler
ii python3-nova 3:30.0.0-0ubuntu1~cloud0 all OpenStack Compute Python 3 libraries
ii python3-novaclient 2:18.7.0-0ubuntu1~cloud0 all client library for OpenStack Compute API - 3.x
-------
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2100342/+subscriptions
References
