← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #93696

[Bug 2056613] [NEW] libvirt CPU power management does not support live migration

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
libvirt CPU power management does not support live migration

Steps to reproduce
==================
1. Turn on libvirt CPU power management
2. Boot an instance with hw:cpu_policy=dedicated
3. Live migrate the instance

Expected result
===============
Live migration succeeds.

Actual result
=============
Live migration fails with the following libvirt error in the source nova-compute logs:

[instance: afdd5e62-2a97-4b58-a7e7-bb92152f4165] Migration operation thread notification {{(pid=103809) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:10668}}
Feb 21 19:21:15.045216 np0036828692 nova-compute[103809]: Traceback (most recent call last):
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     timer()
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     cb(*args, **kw)
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 173, in _do_send
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     waiter.switch(result)
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 264, in main
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     result = function(*args, **kwargs)
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     return func(*args, **kwargs)
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10322, in _live_migration_operation
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     with excutils.save_and_reraise_exception():
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     self.force_reraise()
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     raise self.value
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10311, in _live_migration_operation
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     guest.migrate(self._live_migration_uri(dest),
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 648, in migrate
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     self._domain.migrateToURI3(
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     result = proxy_call(self._autowrap, f, *args, **kwargs)
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     rv = execute(f, *args, **kwargs)
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     raise e.with_traceback(tb)
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     rv = meth(*args, **kwargs)
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/usr/lib/python3/dist-packages/libvirt.py", line 2126, in migrateToURI3
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     raise libvirtError('virDomainMigrateToURI3() failed')
Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]: libvirt.libvirtError: cannot set CPU affinity on process 48279: Invalid argument

Environment
===========
This was originally noticed in a whitebox CI job [1] on devstack master.

Additional info
===============
Regardless of whether NUMA live migration has changed the underlying CPU pinnings, it's necessary to make sure the cores are powered up on the destination, otherwise libvirt attempts to pin the instance to an offline core. Nova doesn't handle that. With some refactoring to the code itself, it's possible to observe the cores not being powered on in functional tests.

[1]
https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs

** Affects: nova
     Importance: Critical
         Status: In Progress

** Changed in: nova
   Importance: Undecided => Critical

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2056613

Title:
   libvirt CPU power management does not support live migration

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========
  libvirt CPU power management does not support live migration

  Steps to reproduce
  ==================
  1. Turn on libvirt CPU power management
  2. Boot an instance with hw:cpu_policy=dedicated
  3. Live migrate the instance

  Expected result
  ===============
  Live migration succeeds.

  Actual result
  =============
  Live migration fails with the following libvirt error in the source nova-compute logs:

  [instance: afdd5e62-2a97-4b58-a7e7-bb92152f4165] Migration operation thread notification {{(pid=103809) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:10668}}
  Feb 21 19:21:15.045216 np0036828692 nova-compute[103809]: Traceback (most recent call last):
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/hub.py", line 471, in fire_timers
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     timer()
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/hubs/timer.py", line 59, in __call__
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     cb(*args, **kw)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/event.py", line 173, in _do_send
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     waiter.switch(result)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/greenthread.py", line 264, in main
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     result = function(*args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/utils.py", line 664, in context_wrapper
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     return func(*args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10322, in _live_migration_operation
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     with excutils.save_and_reraise_exception():
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     self.force_reraise()
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     raise self.value
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 10311, in _live_migration_operation
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     guest.migrate(self._live_migration_uri(dest),
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 648, in migrate
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     self._domain.migrateToURI3(
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 186, in doit
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     result = proxy_call(self._autowrap, f, *args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     rv = execute(f, *args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 125, in execute
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     raise e.with_traceback(tb)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/opt/stack/data/venv/lib/python3.10/site-packages/eventlet/tpool.py", line 82, in tworker
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     rv = meth(*args, **kwargs)
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:   File "/usr/lib/python3/dist-packages/libvirt.py", line 2126, in migrateToURI3
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]:     raise libvirtError('virDomainMigrateToURI3() failed')
  Feb 21 19:21:15.045387 np0036828692 nova-compute[103809]: libvirt.libvirtError: cannot set CPU affinity on process 48279: Invalid argument

  Environment
  ===========
  This was originally noticed in a whitebox CI job [1] on devstack master.

  Additional info
  ===============
  Regardless of whether NUMA live migration has changed the underlying CPU pinnings, it's necessary to make sure the cores are powered up on the destination, otherwise libvirt attempts to pin the instance to an offline core. Nova doesn't handle that. With some refactoring to the code itself, it's possible to observe the cores not being powered on in functional tests.

  [1]
  https://zuul.opendev.org/t/openstack/build/532b30767df54147a01508e7616930f5/logs

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2056613/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next