yahoo-eng-team team mailing list archive

[OpenStack Infra <2046549@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/904183
Committed: https://opendev.org/openstack/nova/commit/e618e78edc6293d248a5fa2eb63b3fa636250fca
Submitter: "Zuul (22348)"
Branch:    master

commit e618e78edc6293d248a5fa2eb63b3fa636250fca
Author: songjie <songjie_yewu@xxxxxxxxxxxxxxxxxxxx>
Date:   Mon Dec 25 16:59:36 2023 +0800

    libvirt: stop enabling hyperv feature reenlightenment
    
    The 'reenlightenment' hyperv enlightenment will cause
    instances live-migration to fail (KVM currently doesn’t
    fully support reenlightenment notifications, see
    www.qemu.org/docs/master/system/i386/hyperv.html),
    so don't enable it now.
    
    Change-Id: I6821819450bc96e4304125ea3b76a0e462e6e33f
    Closes-Bug: #2046549
    Related-Bug: #2009280


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2046549

Title:
  hyperv reenlightenment can break live-migration

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  With ZED nova enabled some new hyperv settings for instances with
  os_type="windows" property (https://opendev.org/openstack/nova/commit/57ab45323cf5617ebd2decd757e708673d949a8f).

  Enabling the reenlightenment setting can lead to unmigrateable VMs.
  QEMU currently does not support full re-enlightenment notifications (see https://www.qemu.org/docs/master/system/i386/hyperv.html).
  Windows VMs which have hyperv or WSL enabled make use of that feature,
  but qemu can only migrate them if a tsc-frequency is set. The frequency
  setting however is not applied and when set would restrict the target hv
  to support the exact frequency.

  Trying to migrate instances with reenlightenment enabled will fail

  Traceback (most recent call last):
    File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/hub.py", line 476, in fire_timers
      timer()                                                                                                                                              File "/usr/local/lib/python3.10/dist-packages/eventlet/hubs/timer.py", line 59, in __call__
      cb(*args, **kw)
    File "/usr/local/lib/python3.10/dist-packages/eventlet/event.py", line 175, in _do_send
      waiter.switch(result)
    File "/usr/local/lib/python3.10/dist-packages/eventlet/greenthread.py", line 221, in main
      result = function(*args, **kwargs)                                                                                                                   File "/usr/local/lib/python3.10/dist-packages/nova/utils.py", line 656, in context_wrapper
      return func(*args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/driver.py", line 10214, in _live_migration_operation
      with excutils.save_and_reraise_exception():
    File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
      self.force_reraise()
    File "/usr/local/lib/python3.10/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
      raise self.value
    File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/driver.py", line 10203, in _live_migration_operation
      guest.migrate(self._live_migration_uri(dest),
    File "/usr/local/lib/python3.10/dist-packages/nova/virt/libvirt/guest.py", line 642, in migrate
      self._domain.migrateToURI3(
    File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 193, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 151, in proxy_call                                                                rv = execute(f, *args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 132, in execute
      six.reraise(c, e, tb)
    File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
      raise value
    File "/usr/local/lib/python3.10/dist-packages/eventlet/tpool.py", line 86, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/libvirt.py", line 2157, in migrateToURI3
      raise libvirtError('virDomainMigrateToURI3() failed')
  libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 
  2023-12-14T09:05:01.441735Z qemu-system-x86_64: Guest enabled re-enlightenment notifications, 'tsc-frequency=' has to be specified
  2023-12-14T09:05:01.441854Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'cpu'
  2023-12-14T09:05:01.448766Z qemu-system-x86_64: load of migration failed: Invalid argument

  
  Adding the tsc clock or removing the hypver reenlightenment will make migration work again.

  eg.:
     <clock>
        <timer name='tsc' frequency='2095078000'/>
      </clock>

  
  A related bug for windows enlightenments targets evmcs: https://bugs.launchpad.net/nova/+bug/2009280

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2046549/+subscriptions