yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2056612] [NEW] libvirt CPU power management does not handle `isolate` emulator thread policy

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Artom Lifshitz <2056612@xxxxxxxxxxxxxxxxxx>
Date: Sat, 09 Mar 2024 01:30:58 -0000
Reply-to: Bug 2056612 <2056612@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Description
===========
libvirt CPU power management does not handle `isolate` emulator thread policy.

Steps to reproduce
==================
1. Turn on libvirt CPU power management
2. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate

Expected result
===============
After the execution of the steps above, what should have
happened if the issue wasn't present?

Actual result
=============
The instance doesn't start, with the following libvirt error in the nova-compute log:

Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: : libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Traceback (most recent call last):
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     return self._domain.createWithFlags(flags)
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     result = proxy_call(self._autowrap, f, *args, **kwargs)
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     rv = execute(f, *args, **kwargs)
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     raise e.with_traceback(tb)
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     rv = meth(*args, **kwargs)
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags
Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     raise libvirtError('virDomainCreateWithFlags() failed')
Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument
Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest
Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.driver [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] [instance: f697a24e-6599-4ec0-9e3b-87eba1a81a0b] Failed to start libvirt guest: libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument

Environment
===========
This was originally noticed in a whitebox CI job [1] on devstack master.

Additional info
===============
When powering up an instance's CPU, Nova doesn't take into account that with the `isolate` emulator thread policy, there's an extra CPU being consumed by the emulator thread. In a real deployment, this results in libvirt trying to pin the instance to an offline CPU. In functional tests, it's relatively easy to observe that CPU not being powered on.

[1]
https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs

** Affects: nova
     Importance: Undecided
         Status: In Progress

** Description changed:

  Description
  ===========
  libvirt CPU power management does not handle `isolate` emulator thread policy.
  
  Steps to reproduce
  ==================
- 1. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate
- 
+ 1. Turn on libvirt CPU power management
+ 2. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate
  
  Expected result
  ===============
  After the execution of the steps above, what should have
  happened if the issue wasn't present?
  
  Actual result
  =============
  The instance doesn't start, with the following libvirt error in the nova-compute log:
  
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: : libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Traceback (most recent call last):
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     return self._domain.createWithFlags(flags)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     rv = execute(f, *args, **kwargs)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     raise e.with_traceback(tb)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     rv = meth(*args, **kwargs)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     raise libvirtError('virDomainCreateWithFlags() failed')
  Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument
- Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest 
+ Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest
  Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.driver [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] [instance: f697a24e-6599-4ec0-9e3b-87eba1a81a0b] Failed to start libvirt guest: libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument
  
  Environment
  ===========
  This was originally noticed in a whitebox CI job [1] on devstack master.
  
  Additional info
  ===============
  When powering up an instance's CPU, Nova doesn't take into account that with the `isolate` emulator thread policy, there's an extra CPU being consumed by the emulator thread. In a real deployment, this results in libvirt trying to pin the instance to an offline CPU. In functional tests, it's relatively easy to observe that CPU not being powered on.
  
  [1]
  https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2056612

Title:
  libvirt CPU power management does not handle `isolate` emulator thread
  policy

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========
  libvirt CPU power management does not handle `isolate` emulator thread policy.

  Steps to reproduce
  ==================
  1. Turn on libvirt CPU power management
  2. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate

  Expected result
  ===============
  After the execution of the steps above, what should have
  happened if the issue wasn't present?

  Actual result
  =============
  The instance doesn't start, with the following libvirt error in the nova-compute log:

  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: : libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Traceback (most recent call last):
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     return self._domain.createWithFlags(flags)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     rv = execute(f, *args, **kwargs)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     raise e.with_traceback(tb)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     rv = meth(*args, **kwargs)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     raise libvirtError('virDomainCreateWithFlags() failed')
  Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument
  Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest
  Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.driver [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] [instance: f697a24e-6599-4ec0-9e3b-87eba1a81a0b] Failed to start libvirt guest: libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument

  Environment
  ===========
  This was originally noticed in a whitebox CI job [1] on devstack master.

  Additional info
  ===============
  When powering up an instance's CPU, Nova doesn't take into account that with the `isolate` emulator thread policy, there's an extra CPU being consumed by the emulator thread. In a real deployment, this results in libvirt trying to pin the instance to an offline CPU. In functional tests, it's relatively easy to observe that CPU not being powered on.

  [1]
  https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2056612/+subscriptions
Follow ups

[Bug 2056612] Re: libvirt CPU power management does not handle `isolate` emulator thread policy
From: OpenStack Infra, 2024-03-13