yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #79846
[Bug 1842616] Re: NUMA vcpus not correctly allocated against numa regions
Couple of things here:
You're not using hw:cpu_policy=dedicated, which means multiple instances
vCPUS can be pinned to the same host CPUs. This explains why they're all
landing on the same NUMA node.
You're not setting hw:mem_page_size, which means Nova does not do per-
NUMA-cell accounting of memory. This also explains why all your
instances are landing on the same NUMA node.
I believe if you set one or the other, or both, you should get the
behaviour you desire. I'm going to close this as Invalid for now, but by
all means feel free to come back with more info if I got something
wrong.
** Changed in: nova
Status: New => Invalid
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1842616
Title:
NUMA vcpus not correctly allocated against numa regions
Status in OpenStack Compute (nova):
Invalid
Bug description:
Hello,
Using Rocky. Ubuntu 18. 2:18.2.1-0ubuntu1~cloud4
CPU overallocation 2.0
RAM overallocation 1.0
2 sockets, 2 numa nodes per socket. HT disabled.
Intended behavior: Creating an instance that has the size of a numa
node and half of the ram of the numa node. Essentially in a full
hypervisor there will be 8 instances, 2 each numa nodes, with their
core contained in the numa node.
Actual behavior: All instances are spinning up on the first numa node,
not respecting neither the ram allocation or the cores allocation.
After two instances that spin up in a numa node, there should not be
any more ram available for a third one, though it gets allocated
anyway.
In the schedulers, NUMATopologyFilter filters are enabled.
FLAVOR DETAILS:
ubuntu@gateway:~$ openstack flavor show r1.V100-4C
+----------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| access_project_ids | 2bf7e287aba345c1a3d0a8e177aeccd6, 67966a3eab174fbc9534161b71252499, 74d26c45f1ea4b64971d7cbfa67d151e, 858743e80de542918fe081bbd57c5145 |
| disk | 80 |
| id | 1e55637f-bb09-447c-84bc-181931a36e50 |
| name | r1.V100-4C |
| os-flavor-access:is_public | False |
| properties | hw:cpu_cores='7', hw:cpu_sockets='1', hw:numa_nodes='1', vgpu-type='hpe-r6-vgpu-V100-4C' |
| ram | 46080 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 7 |
+----------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
## NUMA
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6
node 0 size: 96419 MB
node 0 free: 86437 MB
node 1 cpus: 7 8 9 10 11 12 13
node 1 size: 96745 MB
node 1 free: 95438 MB
node 2 cpus: 14 15 16 17 18 19 20
node 2 size: 96766 MB
node 2 free: 92110 MB
node 3 cpus: 21 22 23 24 25 26 27
node 3 size: 96766 MB
node 3 free: 96498 MB
node distances:
node 0 1 2 3
0: 10 21 31 31
1: 21 10 31 31
2: 31 31 10 21
3: 31 31 21 10
## INSTANCES details (spun up one after the other)
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh list --all
Id Name State
----------------------------------------------------
1 instance-00008c77 running
2 instance-00008c7a running
3 instance-00008c7d running
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c77
VCPU: 0
CPU: 3
State: running
CPU time: 14.1s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 4
State: running
CPU time: 0.9s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 0
State: running
CPU time: 1.3s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 2
State: running
CPU time: 2.0s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 4
State: running
CPU time: 1.8s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 1
State: running
CPU time: 1.1s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 2
State: running
CPU time: 1.1s
CPU Affinity: yyyyyyy---------------------
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c7a
VCPU: 0
CPU: 4
State: running
CPU time: 14.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 1
State: running
CPU time: 0.9s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 4
State: running
CPU time: 1.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 3
State: running
CPU time: 0.8s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 4
State: running
CPU time: 1.7s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 3
State: running
CPU time: 3.6s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 2
State: running
CPU time: 1.4s
CPU Affinity: yyyyyyy---------------------
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c7a
VCPU: 0
CPU: 6
State: running
CPU time: 14.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 2
State: running
CPU time: 0.9s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 0
State: running
CPU time: 1.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 3
State: running
CPU time: 0.8s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 6
State: running
CPU time: 1.7s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 1
State: running
CPU time: 3.6s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 4
State: running
CPU time: 1.4s
CPU Affinity: yyyyyyy---------------------
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c7d
VCPU: 0
CPU: 4
State: running
CPU time: 13.7s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 4
State: running
CPU time: 1.4s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 6
State: running
CPU time: 1.0s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 5
State: running
CPU time: 0.5s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 0
State: running
CPU time: 0.5s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 1
State: running
CPU time: 0.3s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 2
State: running
CPU time: 1.2s
CPU Affinity: yyyyyyy---------------------
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1842616/+subscriptions
References
