yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2022955] [NEW] FileNotFound when offlining a core due to a privsep context missing

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sylvain Bauza <2022955@xxxxxxxxxxxxxxxxxx>
Date: Mon, 05 Jun 2023 16:21:11 -0000
Reply-to: Bug 2022955 <2022955@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

When we created the CPU power interface, we forgot to add a specific privsep decorator for the set_offline() method :
https://review.opendev.org/c/openstack/nova/+/868236/5/nova/virt/libvirt/cpu/core.py#63

As a result, we have a FileNotFound due to a permission error when
restarting the nova-compute service :

Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service Traceback (most recent call last):
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/filesystem.py", line 56, in write_sys
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     with open(os.path.join(SYS, path), mode='w') as fd:
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service PermissionError: [Errno 13] Permission denied: '/sys/devices/system/cpu/cpu1/online'
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service 
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service The above exception was the direct cause of the following exception:
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service 
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service Traceback (most recent call last):
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/usr/local/lib/python3.10/dist-packages/oslo_service/service.py", line 806, in run_service
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     service.start()
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 162, in start
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     self.manager.init_host(self.service_ref)
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1608, in init_host
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     self.driver.init_host(host=self.host)
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 831, in init_host
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     libvirt_cpu.power_down_all_dedicated_cpus()
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/cpu/api.py", line 128, in power_down_all_dedicated_cpus
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     pcpu.online = False
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/cpu/api.py", line 50, in online
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     core.set_offline(self.ident)
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/cpu/core.py", line 64, in set_offline
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     filesystem.write_sys(os.path.join(gen_cpu_path(core), 'online'), data='0')
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/filesystem.py", line 59, in write_sys
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     raise exception.FileNotFound(file_path=path) from exc
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service nova.exception.FileNotFound: File /sys/devices/system/cpu/cpu1/online could not be found.
Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service

** Affects: nova
     Importance: Undecided
     Assignee: Sylvain Bauza (sylvain-bauza)
         Status: In Progress


** Tags: cpu libvirt

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2022955

Title:
  FileNotFound when offlining a core due to a privsep context missing

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  When we created the CPU power interface, we forgot to add a specific privsep decorator for the set_offline() method :
  https://review.opendev.org/c/openstack/nova/+/868236/5/nova/virt/libvirt/cpu/core.py#63

  As a result, we have a FileNotFound due to a permission error when
  restarting the nova-compute service :

  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service Traceback (most recent call last):
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/filesystem.py", line 56, in write_sys
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     with open(os.path.join(SYS, path), mode='w') as fd:
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service PermissionError: [Errno 13] Permission denied: '/sys/devices/system/cpu/cpu1/online'
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service 
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service The above exception was the direct cause of the following exception:
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service 
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service Traceback (most recent call last):
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/usr/local/lib/python3.10/dist-packages/oslo_service/service.py", line 806, in run_service
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     service.start()
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/service.py", line 162, in start
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     self.manager.init_host(self.service_ref)
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/compute/manager.py", line 1608, in init_host
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     self.driver.init_host(host=self.host)
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 831, in init_host
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     libvirt_cpu.power_down_all_dedicated_cpus()
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/cpu/api.py", line 128, in power_down_all_dedicated_cpus
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     pcpu.online = False
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/cpu/api.py", line 50, in online
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     core.set_offline(self.ident)
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/virt/libvirt/cpu/core.py", line 64, in set_offline
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     filesystem.write_sys(os.path.join(gen_cpu_path(core), 'online'), data='0')
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service   File "/opt/stack/nova/nova/filesystem.py", line 59, in write_sys
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service     raise exception.FileNotFound(file_path=path) from exc
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service nova.exception.FileNotFound: File /sys/devices/system/cpu/cpu1/online could not be found.
  Jun 05 16:18:49 sbauza-dev2 nova-compute[76374]: ERROR oslo_service.service

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2022955/+subscriptions
Follow ups

[Bug 2022955] Re: FileNotFound when offlining a core due to a privsep context missing
From: OpenStack Infra, 2023-06-07