← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #93538

[Bug 2054384] [NEW] nova instance taking shared cpus while flavor has hw_cpu_policy=dedicated configuration

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

when I am trying to use dedicated cpus, it is taking from shared cpu
list only...

1. I have 4 shared and 44 dedicated cpus in the compute.

root@computedp:/home/cdot# cat /etc/nova/nova.conf 
cpu_shared_set = 0-3
cpu_dedicated_set = 4-47


2. I am just trying to assign cpus from dedicated list in my instance, using below flavor for that
root@controller# openstack flavor show testnew
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| access_project_ids         | None                                 |
| description                | None                                 |
| disk                       | 10                                   |
| id                         | 37f816c6-56d7-4e16-8253-75de7d64690e |
| name                       | testnew                              |
| os-flavor-access:is_public | True                                 |
| properties                 | hw:cpu_policy='dedicated'            |
| ram                        | 8192                                 |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 4                                    |
+----------------------------+--------------------------------------+


3. nova-scheduler log is showing

2024-02-20 11:14:08.447 3247 INFO nova.virt.hardware
[req-a5c01121-1224-4346-876d-d43c23b3c4b0
0adba34df2344b218b0ecd329a5464b8 f665d61d52ff4f2d963a4321b12a5842 -
default default] Computed NUMA topology CPU pinning: usable pCPUs: [[2],
[8], [14], [20], [26], [32], [38], [44], [4], [10], [16], [22], [28],
[34], [40], [46], [0], [6], [12], [18], [24], [30], [36], [42]], vCPUs
mapping: [(0, 2), (1, 8), (2, 14), (3, 20)]

4. instance is launched and taking cpus from shared list only
root@computedp:# virsh vcpupin 37
 VCPU   CPU Affinity
----------------------
 0      0,2
 1      0,2
 2      0,2
 3      0,2

5. instance log also doesn't contain any dedicated info
root@computedp:# cat /var/log/libvirt/qemu/instance-00000085.log 
2024-02-20 05:44:10.539+0000: starting up libvirt version: 6.0.0, package: 0ubuntu8.16 (Marc Deslauriers <marc.deslauriers@xxxxxxxxxx> Wed, 20 Apr 2022 11:31:12 -0400), qemu version: 4.2.1Debian 1:4.2-3ubuntu6.28, kernel: 5.4.0-125-generic, hostname: computedp
LC_ALL=C \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-37-instance-00000085 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-37-instance-00000085/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-37-instance-00000085/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-37-instance-00000085/.config \
QEMU_AUDIO_DRV=none \
/usr/bin/qemu-system-x86_64 \
-name guest=instance-00000085,debug-threads=on \
-S \
-object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-37-instance-00000085/master-key.aes \
-machine pc-i440fx-4.2,accel=kvm,usb=off,dump-guest-core=off \
-cpu host \
-m 8192 \
-overcommit mem-lock=off \
-smp 4,sockets=4,cores=1,threads=1 \
-object memory-backend-ram,id=ram-node0,size=8589934592,host-nodes=0,policy=bind \
-numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \
-uuid 1ae64fb2-48d6-4882-b043-73411f026c79 \
-smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=23.2.1,serial=1ae64fb2-48d6-4882-b043-73411f026c79,uuid=1ae64fb2-48d6-4882-b043-73411f026c79,family=Virtual Machine' \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/e6b21f0cb7f233629d4fde570bccaffae2b32448","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/nova/instances/1ae64fb2-48d6-4882-b043-73411f026c79/disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \
-device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on \
-netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=40 \
-device virtio-net-pci,host_mtu=1450,netdev=hostnet0,id=net0,mac=fa:16:3e:13:60:f6,bus=pci.0,addr=0x3 \
-add-fd set=3,fd=42 \
-chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on \
-device isa-serial,chardev=charserial0,id=serial0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:5 \
-device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
-object rng-random,id=objrng0,filename=/dev/urandom \
-device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2024-02-20 05:44:10.539+0000: Domain id=37 is tainted: host-cpu
char device redirected to /dev/pts/19 (label charserial0)

6. Although irrelevant but my system also has isolated cpus as well..
root@computedp:# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-5.4.0-125-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro isolcpus=0-43 nohz_full=0-43 default_hugepagesz=1G hugepagesz=1G hugepages=500 hugepagesz=2M hugepages=1024

Let me know if you find any issue with the setup or is it an expected
behavior?

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2054384

Title:
  nova instance taking shared cpus while flavor has
  hw_cpu_policy=dedicated configuration

Status in OpenStack Compute (nova):
  New

Bug description:
  when I am trying to use dedicated cpus, it is taking from shared cpu
  list only...

  1. I have 4 shared and 44 dedicated cpus in the compute.

  root@computedp:/home/cdot# cat /etc/nova/nova.conf 
  cpu_shared_set = 0-3
  cpu_dedicated_set = 4-47

  
  2. I am just trying to assign cpus from dedicated list in my instance, using below flavor for that
  root@controller# openstack flavor show testnew
  +----------------------------+--------------------------------------+
  | Field                      | Value                                |
  +----------------------------+--------------------------------------+
  | OS-FLV-DISABLED:disabled   | False                                |
  | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
  | access_project_ids         | None                                 |
  | description                | None                                 |
  | disk                       | 10                                   |
  | id                         | 37f816c6-56d7-4e16-8253-75de7d64690e |
  | name                       | testnew                              |
  | os-flavor-access:is_public | True                                 |
  | properties                 | hw:cpu_policy='dedicated'            |
  | ram                        | 8192                                 |
  | rxtx_factor                | 1.0                                  |
  | swap                       |                                      |
  | vcpus                      | 4                                    |
  +----------------------------+--------------------------------------+

  
  3. nova-scheduler log is showing

  2024-02-20 11:14:08.447 3247 INFO nova.virt.hardware
  [req-a5c01121-1224-4346-876d-d43c23b3c4b0
  0adba34df2344b218b0ecd329a5464b8 f665d61d52ff4f2d963a4321b12a5842 -
  default default] Computed NUMA topology CPU pinning: usable pCPUs:
  [[2], [8], [14], [20], [26], [32], [38], [44], [4], [10], [16], [22],
  [28], [34], [40], [46], [0], [6], [12], [18], [24], [30], [36], [42]],
  vCPUs mapping: [(0, 2), (1, 8), (2, 14), (3, 20)]

  4. instance is launched and taking cpus from shared list only
  root@computedp:# virsh vcpupin 37
   VCPU   CPU Affinity
  ----------------------
   0      0,2
   1      0,2
   2      0,2
   3      0,2

  5. instance log also doesn't contain any dedicated info
  root@computedp:# cat /var/log/libvirt/qemu/instance-00000085.log 
  2024-02-20 05:44:10.539+0000: starting up libvirt version: 6.0.0, package: 0ubuntu8.16 (Marc Deslauriers <marc.deslauriers@xxxxxxxxxx> Wed, 20 Apr 2022 11:31:12 -0400), qemu version: 4.2.1Debian 1:4.2-3ubuntu6.28, kernel: 5.4.0-125-generic, hostname: computedp
  LC_ALL=C \
  PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
  HOME=/var/lib/libvirt/qemu/domain-37-instance-00000085 \
  XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-37-instance-00000085/.local/share \
  XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-37-instance-00000085/.cache \
  XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-37-instance-00000085/.config \
  QEMU_AUDIO_DRV=none \
  /usr/bin/qemu-system-x86_64 \
  -name guest=instance-00000085,debug-threads=on \
  -S \
  -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-37-instance-00000085/master-key.aes \
  -machine pc-i440fx-4.2,accel=kvm,usb=off,dump-guest-core=off \
  -cpu host \
  -m 8192 \
  -overcommit mem-lock=off \
  -smp 4,sockets=4,cores=1,threads=1 \
  -object memory-backend-ram,id=ram-node0,size=8589934592,host-nodes=0,policy=bind \
  -numa node,nodeid=0,cpus=0-3,memdev=ram-node0 \
  -uuid 1ae64fb2-48d6-4882-b043-73411f026c79 \
  -smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=23.2.1,serial=1ae64fb2-48d6-4882-b043-73411f026c79,uuid=1ae64fb2-48d6-4882-b043-73411f026c79,family=Virtual Machine' \
  -no-user-config \
  -nodefaults \
  -chardev socket,id=charmonitor,fd=32,server,nowait \
  -mon chardev=charmonitor,id=monitor,mode=control \
  -rtc base=utc,driftfix=slew \
  -global kvm-pit.lost_tick_policy=delay \
  -no-hpet \
  -no-shutdown \
  -boot strict=on \
  -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
  -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/_base/e6b21f0cb7f233629d4fde570bccaffae2b32448","node-name":"libvirt-2-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
  -blockdev '{"node-name":"libvirt-2-format","read-only":true,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
  -blockdev '{"driver":"file","filename":"/var/lib/nova/instances/1ae64fb2-48d6-4882-b043-73411f026c79/disk","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
  -blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":"libvirt-2-format"}' \
  -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=libvirt-1-format,id=virtio-disk0,bootindex=1,write-cache=on \
  -netdev tap,fd=39,id=hostnet0,vhost=on,vhostfd=40 \
  -device virtio-net-pci,host_mtu=1450,netdev=hostnet0,id=net0,mac=fa:16:3e:13:60:f6,bus=pci.0,addr=0x3 \
  -add-fd set=3,fd=42 \
  -chardev pty,id=charserial0,logfile=/dev/fdset/3,logappend=on \
  -device isa-serial,chardev=charserial0,id=serial0 \
  -device usb-tablet,id=input0,bus=usb.0,port=1 \
  -vnc 0.0.0.0:5 \
  -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 \
  -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 \
  -object rng-random,id=objrng0,filename=/dev/urandom \
  -device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.0,addr=0x6 \
  -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
  -msg timestamp=on
  2024-02-20 05:44:10.539+0000: Domain id=37 is tainted: host-cpu
  char device redirected to /dev/pts/19 (label charserial0)

  6. Although irrelevant but my system also has isolated cpus as well..
  root@computedp:# cat /proc/cmdline 
  BOOT_IMAGE=/vmlinuz-5.4.0-125-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro isolcpus=0-43 nohz_full=0-43 default_hugepagesz=1G hugepagesz=1G hugepages=500 hugepagesz=2M hugepages=1024

  Let me know if you find any issue with the setup or is it an expected
  behavior?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2054384/+subscriptions
  Thread Previous • Date Previous • • Thread Next