← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #93218

[Bug 2046549] [NEW] hyperv reenlightenment can break live-migration

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

With ZED nova enabled some new hyperv settings for instances with
os_type="windows" property (https://opendev.org/openstack/nova/commit/57ab45323cf5617ebd2decd757e708673d949a8f).

Enabling the reenlightenment setting can lead to unmigrateable VMs.
QEMU currently does not support full re-enlightenment notifications (see https://www.qemu.org/docs/master/system/i386/hyperv.html).
Windows VMs which have hyperv or WSL enabled make use of that feature,
but qemu can only migrate them if a tsc-frequency is set. The frequency
setting however is not applied and when set would restrict the target hv
to support the exact frequency.

Trying to migrate instances with reenlightenment enabled will fail

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers
    timer()                                                                                                                                              File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/timer.py", line 59, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python3.10/dist-packages/eventlet/event.py", line 175, in _do_send
    waiter.switch(result)
  File "/usr/local/lib/python3.10/dist-packages/eventlet/greenthread.py", line 221, in main
    result = function(*args, **kwargs)                                                                                                                   File "/usr/local/lib/python3.10/dist-packages/nova/utils.py", line 656, in context_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/driver.py", line 10214, in _live_migration_operation
    with excutils.save_and_reraise_exception():
  File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
    self.force_reraise()
  File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
    raise self.value
  File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/driver.py", line 10203, in _live_migration_operation
    guest.migrate(self._live_migration_uri(dest),
  File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/guest.py", line 642, in migrate
    self._domain.migrateToURI3(
  File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 193, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 151, in proxy_call                                                                rv = execute(f, *args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 132, in execute
    six.reraise(c, e, tb)
  File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
    raise value
  File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 86, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/libvirt.py", line 2157, in migrateToURI3
    raise libvirtError('virDomainMigrateToURI3() failed')
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 
2023-12-14T09:05:01.441735Z qemu-system-x86_64: Guest enabled re-enlightenment notifications, 'tsc-frequency=' has to be specified
2023-12-14T09:05:01.441854Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'cpu'
2023-12-14T09:05:01.448766Z qemu-system-x86_64: load of migration failed: Invalid argument


Adding the tsc clock or removing the hypver reenlightenment will make migration work again.

eg.:
   <clock>
      <timer name='tsc' frequency='2095078000'/>
    </clock>


A related bug for windows enlightenments targets evmcs: https://bugs.launchpad.net/nova/+bug/2009280

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: libvirt windows

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2046549

Title:
  hyperv reenlightenment can break live-migration

Status in OpenStack Compute (nova):
  New

Bug description:
  With ZED nova enabled some new hyperv settings for instances with
  os_type="windows" property (https://opendev.org/openstack/nova/commit/57ab45323cf5617ebd2decd757e708673d949a8f).

  Enabling the reenlightenment setting can lead to unmigrateable VMs.
  QEMU currently does not support full re-enlightenment notifications (see https://www.qemu.org/docs/master/system/i386/hyperv.html).
  Windows VMs which have hyperv or WSL enabled make use of that feature,
  but qemu can only migrate them if a tsc-frequency is set. The frequency
  setting however is not applied and when set would restrict the target hv
  to support the exact frequency.

  Trying to migrate instances with reenlightenment enabled will fail

  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers
      timer()                                                                                                                                              File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/timer.py", line 59, in __call__
      cb(*args, **kw)
    File "/usr/local/lib/python3.10/dist-packages/eventlet/event.py", line 175, in _do_send
      waiter.switch(result)
    File "/usr/local/lib/python3.10/dist-packages/eventlet/greenthread.py", line 221, in main
      result = function(*args, **kwargs)                                                                                                                   File "/usr/local/lib/python3.10/dist-packages/nova/utils.py", line 656, in context_wrapper
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/driver.py", line 10214, in _live_migration_operation
      with excutils.save_and_reraise_exception():
    File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/driver.py", line 10203, in _live_migration_operation
      guest.migrate(self._live_migration_uri(dest),
    File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/guest.py", line 642, in migrate
      self._domain.migrateToURI3(
    File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 193, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 151, in proxy_call                                                                rv = execute(f, *args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 132, in execute
      six.reraise(c, e, tb)
    File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
      raise value
    File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 86, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/libvirt.py", line 2157, in migrateToURI3
      raise libvirtError('virDomainMigrateToURI3() failed')
  libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 
  2023-12-14T09:05:01.441735Z qemu-system-x86_64: Guest enabled re-enlightenment notifications, 'tsc-frequency=' has to be specified
  2023-12-14T09:05:01.441854Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'cpu'
  2023-12-14T09:05:01.448766Z qemu-system-x86_64: load of migration failed: Invalid argument

  
  Adding the tsc clock or removing the hypver reenlightenment will make migration work again.

  eg.:
     <clock>
        <timer name='tsc' frequency='2095078000'/>
      </clock>

  
  A related bug for windows enlightenments targets evmcs: https://bugs.launchpad.net/nova/+bug/2009280

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2046549/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next