yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2045966] [NEW] cpu_power_management_strategy=cpu_state failes if the host supports no governors

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Balazs Gibizer <2045966@xxxxxxxxxxxxxxxxxx>
Date: Fri, 08 Dec 2023 11:09:24 -0000
Reply-to: Bug 2045966 <2045966@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

The nova-compute service with libvirt driver fails to start with power
management enabled with cpu_startegy if the host OS only support
offlining CPUs but does not support any cpufreq governor.

The cpu_state strategy should not depend on any cpufreq governor so
nova-compute should start.


[root@edpm-compute-0 ~]# egrep -e 'cpu_dedicated|power|strategy' /var/lib/openstack/config/nova/ -R
/var/lib/openstack/config/nova/01-nova.conf:cpu_power_management=true
/var/lib/openstack/config/nova/05-nova-extra.conf:cpu_dedicated_set = 1


2023-12-08 09:54:43.865 2 ERROR oslo_service.service [None req-7109fd97-f101-452e-abcd-37c0b52b649e - - - - - -] Error starting thread.: nova.exception.FileNotFound: File /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor could not be found.
2023-12-08 09:54:43.865 2 ERROR oslo_service.service Traceback (most recent call last):
2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/filesystem.py", line 37, in read_sys
2023-12-08 09:54:43.865 2 ERROR oslo_service.service     with open(os.path.join(SYS, path), mode='r') as data:
2023-12-08 09:54:43.865 2 ERROR oslo_service.service FileNotFoundError: [Errno 2] No such file or directory: '/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor'
2023-12-08 09:54:43.865 2 ERROR oslo_service.service 
2023-12-08 09:54:43.865 2 ERROR oslo_service.service The above exception was the direct cause of the following exception:
2023-12-08 09:54:43.865 2 ERROR oslo_service.service 
2023-12-08 09:54:43.865 2 ERROR oslo_service.service Traceback (most recent call last):
2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/oslo_service/service.py", line 806, in run_service
2023-12-08 09:54:43.865 2 ERROR oslo_service.service     service.start()
2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/service.py", line 162, in start
2023-12-08 09:54:43.865 2 ERROR oslo_service.service     self.manager.init_host(self.service_ref)
2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 1608, in init_host
2023-12-08 09:54:43.865 2 ERROR oslo_service.service     self.driver.init_host(host=self.host)
2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 825, in init_host
2023-12-08 09:54:43.865 2 ERROR oslo_service.service     libvirt_cpu.validate_all_dedicated_cpus()
2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/cpu/api.py", line 140, in validate_all_dedicated_cpus
2023-12-08 09:54:43.865 2 ERROR oslo_service.service     governors.add(pcpu.governor)
2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/cpu/api.py", line 63, in governor
2023-12-08 09:54:43.865 2 ERROR oslo_service.service     return core.get_governor(self.ident)
2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/cpu/core.py", line 69, in get_governor
2023-12-08 09:54:43.865 2 ERROR oslo_service.service     return filesystem.read_sys(
2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/filesystem.py", line 40, in read_sys
2023-12-08 09:54:43.865 2 ERROR oslo_service.service     raise exception.FileNotFound(file_path=path) from exc
2023-12-08 09:54:43.865 2 ERROR oslo_service.service nova.exception.FileNotFound: File /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor could not be found.


[root@edpm-compute-0 ~]# ll /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
ls: cannot access '/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor': No such file or directory
[root@edpm-compute-0 ~]# ll /sys/devices/system/cpu/cpu1/cpufreq/
ls: cannot access '/sys/devices/system/cpu/cpu1/cpufreq/': No such file or directory
[root@edpm-compute-0 ~]# ll /sys/devices/system/cpu/cpu1/
total 0
drwxr-xr-x. 6 root root    0 Dec  8 10:01 cache
-r--------. 1 root root 4096 Dec  8 09:21 crash_notes
-r--------. 1 root root 4096 Dec  8 09:21 crash_notes_size
lrwxrwxrwx. 1 root root    0 Dec  8 09:21 driver -> ../../../../bus/cpu/drivers/processor
lrwxrwxrwx. 1 root root    0 Dec  8 09:21 firmware_node -> ../../../LNXSYSTM:00/LNXSYBUS:00/ACPI0010:00/LNXCPU:01
drwxr-xr-x. 2 root root    0 Dec  8 09:21 hotplug
lrwxrwxrwx. 1 root root    0 Dec  8 09:21 node0 -> ../../node/node0
-rw-r--r--. 1 root root 4096 Dec  8 10:01 online
drwxr-xr-x. 2 root root    0 Dec  8 09:21 power
lrwxrwxrwx. 1 root root    0 Dec  8 09:21 subsystem -> ../../../../bus/cpu
drwxr-xr-x. 2 root root    0 Dec  8 10:01 topology
-rw-r--r--. 1 root root 4096 Dec  8 09:21 uevent

[root@edpm-compute-0 ~]# cat /etc/*release*
CentOS Stream release 9
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/";
BUG_REPORT_URL="https://bugzilla.redhat.com/";
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
CentOS Stream release 9
CentOS Stream release 9
cpe:/o:centos:centos:9

[root@edpm-compute-0 ~]# uname -a 
Linux edpm-compute-0 5.14.0-390.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 24 10:44:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

** Affects: nova
     Importance: Medium
     Assignee: Balazs Gibizer (balazs-gibizer)
         Status: In Progress


** Tags: libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2045966

Title:
  cpu_power_management_strategy=cpu_state failes if the host supports no
  governors

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  The nova-compute service with libvirt driver fails to start with power
  management enabled with cpu_startegy if the host OS only support
  offlining CPUs but does not support any cpufreq governor.

  The cpu_state strategy should not depend on any cpufreq governor so
  nova-compute should start.

  
  [root@edpm-compute-0 ~]# egrep -e 'cpu_dedicated|power|strategy' /var/lib/openstack/config/nova/ -R
  /var/lib/openstack/config/nova/01-nova.conf:cpu_power_management=true
  /var/lib/openstack/config/nova/05-nova-extra.conf:cpu_dedicated_set = 1


  2023-12-08 09:54:43.865 2 ERROR oslo_service.service [None req-7109fd97-f101-452e-abcd-37c0b52b649e - - - - - -] Error starting thread.: nova.exception.FileNotFound: File /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor could not be found.
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service Traceback (most recent call last):
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/filesystem.py", line 37, in read_sys
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service     with open(os.path.join(SYS, path), mode='r') as data:
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service FileNotFoundError: [Errno 2] No such file or directory: '/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor'
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service 
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service The above exception was the direct cause of the following exception:
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service 
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service Traceback (most recent call last):
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/oslo_service/service.py", line 806, in run_service
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service     service.start()
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/service.py", line 162, in start
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service     self.manager.init_host(self.service_ref)
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/compute/manager.py", line 1608, in init_host
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service     self.driver.init_host(host=self.host)
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 825, in init_host
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service     libvirt_cpu.validate_all_dedicated_cpus()
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/cpu/api.py", line 140, in validate_all_dedicated_cpus
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service     governors.add(pcpu.governor)
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/cpu/api.py", line 63, in governor
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service     return core.get_governor(self.ident)
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/cpu/core.py", line 69, in get_governor
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service     return filesystem.read_sys(
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service   File "/usr/lib/python3.9/site-packages/nova/filesystem.py", line 40, in read_sys
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service     raise exception.FileNotFound(file_path=path) from exc
  2023-12-08 09:54:43.865 2 ERROR oslo_service.service nova.exception.FileNotFound: File /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor could not be found.

  
  [root@edpm-compute-0 ~]# ll /sys/devices/system/cpu/cpu1/cpufreq/scaling_governor
  ls: cannot access '/sys/devices/system/cpu/cpu1/cpufreq/scaling_governor': No such file or directory
  [root@edpm-compute-0 ~]# ll /sys/devices/system/cpu/cpu1/cpufreq/
  ls: cannot access '/sys/devices/system/cpu/cpu1/cpufreq/': No such file or directory
  [root@edpm-compute-0 ~]# ll /sys/devices/system/cpu/cpu1/
  total 0
  drwxr-xr-x. 6 root root    0 Dec  8 10:01 cache
  -r--------. 1 root root 4096 Dec  8 09:21 crash_notes
  -r--------. 1 root root 4096 Dec  8 09:21 crash_notes_size
  lrwxrwxrwx. 1 root root    0 Dec  8 09:21 driver -> ../../../../bus/cpu/drivers/processor
  lrwxrwxrwx. 1 root root    0 Dec  8 09:21 firmware_node -> ../../../LNXSYSTM:00/LNXSYBUS:00/ACPI0010:00/LNXCPU:01
  drwxr-xr-x. 2 root root    0 Dec  8 09:21 hotplug
  lrwxrwxrwx. 1 root root    0 Dec  8 09:21 node0 -> ../../node/node0
  -rw-r--r--. 1 root root 4096 Dec  8 10:01 online
  drwxr-xr-x. 2 root root    0 Dec  8 09:21 power
  lrwxrwxrwx. 1 root root    0 Dec  8 09:21 subsystem -> ../../../../bus/cpu
  drwxr-xr-x. 2 root root    0 Dec  8 10:01 topology
  -rw-r--r--. 1 root root 4096 Dec  8 09:21 uevent

  [root@edpm-compute-0 ~]# cat /etc/*release*
  CentOS Stream release 9
  NAME="CentOS Stream"
  VERSION="9"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="9"
  PLATFORM_ID="platform:el9"
  PRETTY_NAME="CentOS Stream 9"
  ANSI_COLOR="0;31"
  LOGO="fedora-logo-icon"
  CPE_NAME="cpe:/o:centos:centos:9"
  HOME_URL="https://centos.org/";
  BUG_REPORT_URL="https://bugzilla.redhat.com/";
  REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
  REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
  CentOS Stream release 9
  CentOS Stream release 9
  cpe:/o:centos:centos:9

  [root@edpm-compute-0 ~]# uname -a 
  Linux edpm-compute-0 5.14.0-390.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Nov 24 10:44:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2045966/+subscriptions
Follow ups

[Bug 2045966] Re: cpu_power_management_strategy=cpu_state failes if the host supports no governors
From: OpenStack Infra, 2023-12-08