← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #74355

[Bug 1788028] [NEW] libvirt: nova assumed dest emultor path is the same as source and fails to migrate if this is not true.

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========
live migartion between compute nodes of different distors 
fails due to asstion that the emulator path on dest is 
the same as the source compute node. In past releases
it was acidentally posable to migrate between ubuntu and centos
host because the qemu-kvm binary was packaged in the same location.

in centos 7.5 and ubuntu 16.04 this is nolonger true.


Steps to reproduce
==================
deploy a multinode devestack with one ubuntu node and one centos node.
configure ssh keys and the other normal steps for live migration
and boot a vm on one host. then attempt to live migrate the vm.

Expected result
===============
live migration succeds as it did when the emulator paths were the same.

Actual result
=============

migration fails as the emulator path of the source is used to invoke the 
emulator on the destination node resulting in a file not found error.

Environment
===========
1. Rock RC1 nova sha: afe4512bf66c89a061b1a7ccd3e7ac8e3b1b284d
   the exact version is not important as i belive we have alway
   had the same behavior.
 
2. Which hypervisor did you use?
  libvirt with kvm

2. Which storage type did you use?
  boot form volume with default devstack lvm cinder backend.

3. Which networking type did you use?
   ovs

Logs & Configs
==============

Aug 13 22:04:56 devstack2 nova-compute[1720]: Traceback (most recent call last):
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
Aug 13 22:04:56 devstack2 nova-compute[1720]: timer()
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
Aug 13 22:04:56 devstack2 nova-compute[1720]: cb(*args, **kw)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 168, in _do_send
Aug 13 22:04:56 devstack2 nova-compute[1720]: waiter.switch(result)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
Aug 13 22:04:56 devstack2 nova-compute[1720]: result = function(*args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/utils.py", line 810, in context_wrapper
Aug 13 22:04:56 devstack2 nova-compute[1720]: return func(*args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7134, in _live_migration_operation
Aug 13 22:04:56 devstack2 nova-compute[1720]: LOG.error("Live Migration failure: %s", e, instance=instance)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
Aug 13 22:04:56 devstack2 nova-compute[1720]: self.force_reraise()
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
Aug 13 22:04:56 devstack2 nova-compute[1720]: six.reraise(self.type_, self.value, self.tb)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7127, in _live_migration_operation
Aug 13 22:04:56 devstack2 nova-compute[1720]: bandwidth=CONF.libvirt.live_migration_bandwidth)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 673, in migrate
Aug 13 22:04:56 devstack2 nova-compute[1720]: destination, params=params, flags=flags)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
Aug 13 22:04:56 devstack2 nova-compute[1720]: result = proxy_call(self._autowrap, f, *args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
Aug 13 22:04:56 devstack2 nova-compute[1720]: rv = execute(f, *args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
Aug 13 22:04:56 devstack2 nova-compute[1720]: six.reraise(c, e, tb)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
Aug 13 22:04:56 devstack2 nova-compute[1720]: rv = meth(*args, **kwargs)
Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1745, in migrateToURI3
Aug 13 22:04:56 devstack2 nova-compute[1720]: if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
Aug 13 22:04:56 devstack2 nova-compute[1720]: libvirtError: Cannot check QEMU binary /usr/libexec/qemu-kvm: No such file or directory

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1788028

Title:
  libvirt: nova assumed dest emultor path is the same as source and
  fails to migrate if this is not true.

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  live migartion between compute nodes of different distors 
  fails due to asstion that the emulator path on dest is 
  the same as the source compute node. In past releases
  it was acidentally posable to migrate between ubuntu and centos
  host because the qemu-kvm binary was packaged in the same location.

  in centos 7.5 and ubuntu 16.04 this is nolonger true.

  
  Steps to reproduce
  ==================
  deploy a multinode devestack with one ubuntu node and one centos node.
  configure ssh keys and the other normal steps for live migration
  and boot a vm on one host. then attempt to live migrate the vm.

  Expected result
  ===============
  live migration succeds as it did when the emulator paths were the same.

  Actual result
  =============

  migration fails as the emulator path of the source is used to invoke the 
  emulator on the destination node resulting in a file not found error.

  Environment
  ===========
  1. Rock RC1 nova sha: afe4512bf66c89a061b1a7ccd3e7ac8e3b1b284d
     the exact version is not important as i belive we have alway
     had the same behavior.
   
  2. Which hypervisor did you use?
    libvirt with kvm

  2. Which storage type did you use?
    boot form volume with default devstack lvm cinder backend.

  3. Which networking type did you use?
     ovs

  Logs & Configs
  ==============

  Aug 13 22:04:56 devstack2 nova-compute[1720]: Traceback (most recent call last):
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 457, in fire_timers
  Aug 13 22:04:56 devstack2 nova-compute[1720]: timer()
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 58, in __call__
  Aug 13 22:04:56 devstack2 nova-compute[1720]: cb(*args, **kw)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 168, in _do_send
  Aug 13 22:04:56 devstack2 nova-compute[1720]: waiter.switch(result)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 214, in main
  Aug 13 22:04:56 devstack2 nova-compute[1720]: result = function(*args, **kwargs)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/utils.py", line 810, in context_wrapper
  Aug 13 22:04:56 devstack2 nova-compute[1720]: return func(*args, **kwargs)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7134, in _live_migration_operation
  Aug 13 22:04:56 devstack2 nova-compute[1720]: LOG.error("Live Migration failure: %s", e, instance=instance)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
  Aug 13 22:04:56 devstack2 nova-compute[1720]: self.force_reraise()
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
  Aug 13 22:04:56 devstack2 nova-compute[1720]: six.reraise(self.type_, self.value, self.tb)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7127, in _live_migration_operation
  Aug 13 22:04:56 devstack2 nova-compute[1720]: bandwidth=CONF.libvirt.live_migration_bandwidth)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/opt/stack/nova/nova/virt/libvirt/guest.py", line 673, in migrate
  Aug 13 22:04:56 devstack2 nova-compute[1720]: destination, params=params, flags=flags)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
  Aug 13 22:04:56 devstack2 nova-compute[1720]: result = proxy_call(self._autowrap, f, *args, **kwargs)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
  Aug 13 22:04:56 devstack2 nova-compute[1720]: rv = execute(f, *args, **kwargs)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
  Aug 13 22:04:56 devstack2 nova-compute[1720]: six.reraise(c, e, tb)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
  Aug 13 22:04:56 devstack2 nova-compute[1720]: rv = meth(*args, **kwargs)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1745, in migrateToURI3
  Aug 13 22:04:56 devstack2 nova-compute[1720]: if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
  Aug 13 22:04:56 devstack2 nova-compute[1720]: libvirtError: Cannot check QEMU binary /usr/libexec/qemu-kvm: No such file or directory

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1788028/+subscriptions
  Thread Previous • Date Previous • • Thread Next