yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #79833
[Bug 1842616] [NEW] NUMA vcpus not correctly allocated against numa regions
Public bug reported:
Hello,
Using Rocky. Ubuntu 18. 2:18.2.1-0ubuntu1~cloud4
CPU overallocation 2.0
RAM overallocation 1.0
2 sockets, 2 numa nodes per socket. HT disabled.
Intended behavior: Creating an instance that has the size of a numa node
and half of the ram of the numa node. Essentially in a full hypervisor
there will be 8 instances, 2 each numa nodes, with their core contained
in the numa node.
Actual behavior: All instances are spinning up on the first numa node,
not respecting neither the ram allocation or the cores allocation. After
two instances that spin up in a numa node, there should not be any more
ram available for a third one, though it gets allocated anyway.
In the schedulers, NUMATopologyFilter filters are enabled.
FLAVOR DETAILS:
ubuntu@gateway:~$ openstack flavor show r1.V100-4C
+----------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| access_project_ids | 2bf7e287aba345c1a3d0a8e177aeccd6, 67966a3eab174fbc9534161b71252499, 74d26c45f1ea4b64971d7cbfa67d151e, 858743e80de542918fe081bbd57c5145 |
| disk | 80 |
| id | 1e55637f-bb09-447c-84bc-181931a36e50 |
| name | r1.V100-4C |
| os-flavor-access:is_public | False |
| properties | hw:cpu_cores='7', hw:cpu_sockets='1', hw:numa_nodes='1', vgpu-type='hpe-r6-vgpu-V100-4C' |
| ram | 46080 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 7 |
+----------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
## NUMA
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6
node 0 size: 96419 MB
node 0 free: 86437 MB
node 1 cpus: 7 8 9 10 11 12 13
node 1 size: 96745 MB
node 1 free: 95438 MB
node 2 cpus: 14 15 16 17 18 19 20
node 2 size: 96766 MB
node 2 free: 92110 MB
node 3 cpus: 21 22 23 24 25 26 27
node 3 size: 96766 MB
node 3 free: 96498 MB
node distances:
node 0 1 2 3
0: 10 21 31 31
1: 21 10 31 31
2: 31 31 10 21
3: 31 31 21 10
## INSTANCES details (spun up one after the other)
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh list --all
Id Name State
----------------------------------------------------
1 instance-00008c77 running
2 instance-00008c7a running
3 instance-00008c7d running
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c77
VCPU: 0
CPU: 3
State: running
CPU time: 14.1s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 4
State: running
CPU time: 0.9s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 0
State: running
CPU time: 1.3s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 2
State: running
CPU time: 2.0s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 4
State: running
CPU time: 1.8s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 1
State: running
CPU time: 1.1s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 2
State: running
CPU time: 1.1s
CPU Affinity: yyyyyyy---------------------
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c7a
VCPU: 0
CPU: 4
State: running
CPU time: 14.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 1
State: running
CPU time: 0.9s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 4
State: running
CPU time: 1.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 3
State: running
CPU time: 0.8s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 4
State: running
CPU time: 1.7s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 3
State: running
CPU time: 3.6s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 2
State: running
CPU time: 1.4s
CPU Affinity: yyyyyyy---------------------
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c7a
VCPU: 0
CPU: 6
State: running
CPU time: 14.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 2
State: running
CPU time: 0.9s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 0
State: running
CPU time: 1.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 3
State: running
CPU time: 0.8s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 6
State: running
CPU time: 1.7s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 1
State: running
CPU time: 3.6s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 4
State: running
CPU time: 1.4s
CPU Affinity: yyyyyyy---------------------
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c7d
VCPU: 0
CPU: 4
State: running
CPU time: 13.7s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 4
State: running
CPU time: 1.4s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 6
State: running
CPU time: 1.0s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 5
State: running
CPU time: 0.5s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 0
State: running
CPU time: 0.5s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 1
State: running
CPU time: 0.3s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 2
State: running
CPU time: 1.2s
CPU Affinity: yyyyyyy---------------------
** Affects: nova
Importance: Undecided
Status: New
** Tags: numa scheduler
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1842616
Title:
NUMA vcpus not correctly allocated against numa regions
Status in OpenStack Compute (nova):
New
Bug description:
Hello,
Using Rocky. Ubuntu 18. 2:18.2.1-0ubuntu1~cloud4
CPU overallocation 2.0
RAM overallocation 1.0
2 sockets, 2 numa nodes per socket. HT disabled.
Intended behavior: Creating an instance that has the size of a numa
node and half of the ram of the numa node. Essentially in a full
hypervisor there will be 8 instances, 2 each numa nodes, with their
core contained in the numa node.
Actual behavior: All instances are spinning up on the first numa node,
not respecting neither the ram allocation or the cores allocation.
After two instances that spin up in a numa node, there should not be
any more ram available for a third one, though it gets allocated
anyway.
In the schedulers, NUMATopologyFilter filters are enabled.
FLAVOR DETAILS:
ubuntu@gateway:~$ openstack flavor show r1.V100-4C
+----------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+----------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| access_project_ids | 2bf7e287aba345c1a3d0a8e177aeccd6, 67966a3eab174fbc9534161b71252499, 74d26c45f1ea4b64971d7cbfa67d151e, 858743e80de542918fe081bbd57c5145 |
| disk | 80 |
| id | 1e55637f-bb09-447c-84bc-181931a36e50 |
| name | r1.V100-4C |
| os-flavor-access:is_public | False |
| properties | hw:cpu_cores='7', hw:cpu_sockets='1', hw:numa_nodes='1', vgpu-type='hpe-r6-vgpu-V100-4C' |
| ram | 46080 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 7 |
+----------------------------+----------------------------------------------------------------------------------------------------------------------------------------+
## NUMA
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# numactl --hardware
available: 4 nodes (0-3)
node 0 cpus: 0 1 2 3 4 5 6
node 0 size: 96419 MB
node 0 free: 86437 MB
node 1 cpus: 7 8 9 10 11 12 13
node 1 size: 96745 MB
node 1 free: 95438 MB
node 2 cpus: 14 15 16 17 18 19 20
node 2 size: 96766 MB
node 2 free: 92110 MB
node 3 cpus: 21 22 23 24 25 26 27
node 3 size: 96766 MB
node 3 free: 96498 MB
node distances:
node 0 1 2 3
0: 10 21 31 31
1: 21 10 31 31
2: 31 31 10 21
3: 31 31 21 10
## INSTANCES details (spun up one after the other)
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh list --all
Id Name State
----------------------------------------------------
1 instance-00008c77 running
2 instance-00008c7a running
3 instance-00008c7d running
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c77
VCPU: 0
CPU: 3
State: running
CPU time: 14.1s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 4
State: running
CPU time: 0.9s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 0
State: running
CPU time: 1.3s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 2
State: running
CPU time: 2.0s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 4
State: running
CPU time: 1.8s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 1
State: running
CPU time: 1.1s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 2
State: running
CPU time: 1.1s
CPU Affinity: yyyyyyy---------------------
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c7a
VCPU: 0
CPU: 4
State: running
CPU time: 14.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 1
State: running
CPU time: 0.9s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 4
State: running
CPU time: 1.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 3
State: running
CPU time: 0.8s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 4
State: running
CPU time: 1.7s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 3
State: running
CPU time: 3.6s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 2
State: running
CPU time: 1.4s
CPU Affinity: yyyyyyy---------------------
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c7a
VCPU: 0
CPU: 6
State: running
CPU time: 14.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 2
State: running
CPU time: 0.9s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 0
State: running
CPU time: 1.2s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 3
State: running
CPU time: 0.8s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 6
State: running
CPU time: 1.7s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 1
State: running
CPU time: 3.6s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 4
State: running
CPU time: 1.4s
CPU Affinity: yyyyyyy---------------------
root@compute-r6-h30:/sys/class/mdev_bus/0000:86:00.0# virsh vcpuinfo instance-00008c7d
VCPU: 0
CPU: 4
State: running
CPU time: 13.7s
CPU Affinity: yyyyyyy---------------------
VCPU: 1
CPU: 4
State: running
CPU time: 1.4s
CPU Affinity: yyyyyyy---------------------
VCPU: 2
CPU: 6
State: running
CPU time: 1.0s
CPU Affinity: yyyyyyy---------------------
VCPU: 3
CPU: 5
State: running
CPU time: 0.5s
CPU Affinity: yyyyyyy---------------------
VCPU: 4
CPU: 0
State: running
CPU time: 0.5s
CPU Affinity: yyyyyyy---------------------
VCPU: 5
CPU: 1
State: running
CPU time: 0.3s
CPU Affinity: yyyyyyy---------------------
VCPU: 6
CPU: 2
State: running
CPU time: 1.2s
CPU Affinity: yyyyyyy---------------------
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1842616/+subscriptions
Follow ups
