yahoo-eng-team team mailing list archive

[OpenStack Infra <2056612@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/909795
Committed: https://opendev.org/openstack/nova/commit/0986d2bbe871d852075549eecea46f4b282c0842
Submitter: "Zuul (22348)"
Branch:    master

commit 0986d2bbe871d852075549eecea46f4b282c0842
Author: Artom Lifshitz <alifshit@xxxxxxxxxx>
Date:   Wed Feb 21 16:02:16 2024 -0500

    Power on cores for isolated emulator threads
    
    Previously, with the `isolate` emulator threads policy and libvirt cpu
    power management enabled, we did not power on the cores to which the
    emulator threads were pin. Start doing that, and don't forget to power
    them down when the instance is stopped.
    
    Closes-bug: 2056612
    Change-Id: I6e5383d8a0bf3f0ed8c870754cddae4e9163b4fd


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2056612

Title:
  libvirt CPU power management does not handle `isolate` emulator thread
  policy

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  libvirt CPU power management does not handle `isolate` emulator thread policy.

  Steps to reproduce
  ==================
  1. Turn on libvirt CPU power management
  2. Boot an instance with hw:cpu_policy=dedicated and hw:emulator_threads_policy=isolate

  Expected result
  ===============
  After the execution of the steps above, what should have
  happened if the issue wasn't present?

  Actual result
  =============
  The instance doesn't start, with the following libvirt error in the nova-compute log:

  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: : libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest Traceback (most recent call last):
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 165, in launch
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     return self._domain.createWithFlags(flags)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     rv = execute(f, *args, **kwargs)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     raise e.with_traceback(tb)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     rv = meth(*args, **kwargs)
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest   File "/usr/lib/python3/dist-packages/libvirt.py", line 1385, in createWithFlags
  Feb 21 19:15:31.301470 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest     raise libvirtError('virDomainCreateWithFlags() failed')
  Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument
  Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.guest
  Feb 21 19:15:31.316773 np0036828693 nova-compute[42254]: ERROR nova.virt.libvirt.driver [None req-ec45061f-e9a4-4b02-9354-0cb390bd28cf tempest-EmulatorThreadTest-1184416592 tempest-EmulatorThreadTest-1184416592-project-member] [instance: f697a24e-6599-4ec0-9e3b-87eba1a81a0b] Failed to start libvirt guest: libvirt.libvirtError: cannot set CPU affinity on process 47343: Invalid argument

  Environment
  ===========
  This was originally noticed in a whitebox CI job [1] on devstack master.

  Additional info
  ===============
  When powering up an instance's CPU, Nova doesn't take into account that with the `isolate` emulator thread policy, there's an extra CPU being consumed by the emulator thread. In a real deployment, this results in libvirt trying to pin the instance to an offline CPU. In functional tests, it's relatively easy to observe that CPU not being powered on.

  [1]
  https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2056612/+subscriptions