yahoo-eng-team team mailing list archive

[OpenStack Infra <2056613@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/909806
Committed: https://opendev.org/openstack/nova/commit/c1ccc1a3165ec1556c605b3b036274e992b0a09d
Submitter: "Zuul (22348)"
Branch:    master

commit c1ccc1a3165ec1556c605b3b036274e992b0a09d
Author: Artom Lifshitz <alifshit@xxxxxxxxxx>
Date:   Wed Feb 21 19:58:32 2024 -0500

    pwr mgmt: handle live migrations correctly
    
    Previously, live migrations completely ignored CPU power management.
    This patch makes sure that we correctly:
    
    * Power up the cores on the destination during pre_live_migration, as
      we need them powered up before the instance starts on the
      destination.
    * If the live migration is successful, power down the vacated cores on
      the source.
    * In case of a rollback, power down the cores previously powered up on
      pre_live_migration.
    
    Closes-bug: 2056613
    Change-Id: I787bd7807950370cd865f29b95989d489d4826d0


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2056613

Title:
   libvirt CPU power management does not support live migration

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  libvirt CPU power management does not support live migration

  Steps to reproduce
  ==================
  1. Turn on libvirt CPU power management
  2. Boot an instance with hw:cpu_policy=dedicated
  3. Live migrate the instance

  Expected result
  ===============
  Live migration succeeds.

  Actual result
  =============
  Live migration fails with the following libvirt error in the source nova-compute logs:

  [instance: afdd5e62-2a97-4b58-a7e7-bb92152f4165] Migration operation thread notification {{(pid=103809) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:10668}}
  Feb 21 19:21:15.045216 np0036828692 nova-compute[103809]: Traceback (most recent call last):
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     timer()
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     cb(*args, **kw)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 173, in _do_send
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     waiter.switch(result)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 264, in main
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     result = function(*args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     return func(*args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10322, in _live_migration_operation
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     with excutils.save_and_reraise_exception():
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     self.force_reraise()
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     raise self.value
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10311, in _live_migration_operation
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     guest.migrate(self._live_migration_uri(dest),
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 648, in migrate
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     self._domain.migrateToURI3(
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     rv = execute(f, *args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     raise e.with_traceback(tb)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     rv = meth(*args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/usr/lib/python3/dist-packages/libvirt.py", line 2126, in migrateToURI3
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     raise libvirtError('virDomainMigrateToURI3() failed')
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]: libvirt.libvirtError: cannot set CPU affinity on process 48279: Invalid argument

  Environment
  ===========
  This was originally noticed in a whitebox CI job [1] on devstack master.

  Additional info
  ===============
  Regardless of whether NUMA live migration has changed the underlying CPU pinnings, it's necessary to make sure the cores are powered up on the destination, otherwise libvirt attempts to pin the instance to an offline core. Nova doesn't handle that. With some refactoring to the code itself, it's possible to observe the cores not being powered on in functional tests.

  [1]
  https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2056613/+subscriptions