← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #47902

[Bug 1556126] [NEW] Live migrations from Mitaka to Liberty are broken

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I have an environment consisting of three nodes:

Controller Mitaka, commit id: 59a07f00ad3c527e3b39712220cf9de1e68cd16f
Compute Mitaka, commit id: 59a07f00ad3c527e3b39712220cf9de1e68cd16f
Compute Stable/Liberty, commit id: 184e2552490ecfded61db3cc9ba1cd8d6aac1644

I am able to live migrate VMs from Liberty to Mitaka using this CLI
command:

nova --os-compute-api-version 2.24 live-migrate --block-migrate instance

But when I want to move VM from Mitaka to Liberty I'm ending up with an
error:

ERROR nova.virt.libvirt.driver [req-aea14ab8-204d-4e7f-a591-e81a5b5fde4b admin demo] [instance: d509c4af-2003-4075-9c2f-2c4fbce727ea] Live Migration failure: internal error:
 process exited while connecting to monitor: 2016-03-11T13:36:33.803180Z qemu-system-x86_64: -vnc None:0: Failed to start VNC server on `(null)': address resolution failed for None:5900: Temporary
failure in name resolution

Traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
    waiter.switch(result)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/utils.py", line 1145, in context_wrapper
    return func(*args, **kwargs)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6059, in _live_migration_operation
    instance=instance)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6027, in _live_migration_operation
    CONF.libvirt.live_migration_bandwidth)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
    six.reraise(c, e, tb)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1825, in migrateToURI2
    if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: internal error: process exited while connecting to monitor: 2016-03-11T13:20:07.165053Z qemu-system-x86_64: -vnc None:0: Failed to start VNC server on `(null)': address resolution failed for None:5900: Temporary failure in name resolution

Same happens when VM is volume-backed or on shared storage.

This happens because pre_live_migration, that is executed on destination
(Liberty), returns dict that looks like:

{u'volume': {}, u'serial_listen_addr': u'127.0.0.1',
u'graphics_listen_addrs': {u'vnc': u'10.0.0.3', u'spice': u'127.0.0.1'}}

When it comes back to rpc api of new compute node here:
https://github.com/openstack/nova/blob/a3cf38a3ec0fd57679320688bd815225c2bf053f/nova/compute/rpcapi.py#L680

We just pass this data to .from_legacy_dict() migrate_data object
method, but  it does expect that all this data will be nested under
'pre_live_migration_result' key:
https://github.com/openstack/nova/blob/7832f6ea816b1b79251d06abbf38772894e74e2f/nova/objects/migrate_data.py#L195

Because of this we don't really convert data coming from
pre_live_migration phase and pass migrate_data object with Nones that
are needed for setting up VNC.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: live-migration

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1556126

Title:
  Live migrations from Mitaka to Liberty are broken

Status in OpenStack Compute (nova):
  New

Bug description:
  I have an environment consisting of three nodes:

  Controller Mitaka, commit id: 59a07f00ad3c527e3b39712220cf9de1e68cd16f
  Compute Mitaka, commit id: 59a07f00ad3c527e3b39712220cf9de1e68cd16f
  Compute Stable/Liberty, commit id: 184e2552490ecfded61db3cc9ba1cd8d6aac1644

  I am able to live migrate VMs from Liberty to Mitaka using this CLI
  command:

  nova --os-compute-api-version 2.24 live-migrate --block-migrate
  instance

  But when I want to move VM from Mitaka to Liberty I'm ending up with
  an error:

  ERROR nova.virt.libvirt.driver [req-aea14ab8-204d-4e7f-a591-e81a5b5fde4b admin demo] [instance: d509c4af-2003-4075-9c2f-2c4fbce727ea] Live Migration failure: internal error:
   process exited while connecting to monitor: 2016-03-11T13:36:33.803180Z qemu-system-x86_64: -vnc None:0: Failed to start VNC server on `(null)': address resolution failed for None:5900: Temporary
  failure in name resolution

  Traceback:

  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
      timer()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
      cb(*args, **kw)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
      waiter.switch(result)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "/opt/stack/nova/nova/utils.py", line 1145, in context_wrapper
      return func(*args, **kwargs)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6059, in _live_migration_operation
      instance=instance)
    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 6027, in _live_migration_operation
      CONF.libvirt.live_migration_bandwidth)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
      six.reraise(c, e, tb)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1825, in migrateToURI2
      if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
  libvirtError: internal error: process exited while connecting to monitor: 2016-03-11T13:20:07.165053Z qemu-system-x86_64: -vnc None:0: Failed to start VNC server on `(null)': address resolution failed for None:5900: Temporary failure in name resolution

  Same happens when VM is volume-backed or on shared storage.

  This happens because pre_live_migration, that is executed on
  destination (Liberty), returns dict that looks like:

  {u'volume': {}, u'serial_listen_addr': u'127.0.0.1',
  u'graphics_listen_addrs': {u'vnc': u'10.0.0.3', u'spice':
  u'127.0.0.1'}}

  When it comes back to rpc api of new compute node here:
  https://github.com/openstack/nova/blob/a3cf38a3ec0fd57679320688bd815225c2bf053f/nova/compute/rpcapi.py#L680

  We just pass this data to .from_legacy_dict() migrate_data object
  method, but  it does expect that all this data will be nested under
  'pre_live_migration_result' key:
  https://github.com/openstack/nova/blob/7832f6ea816b1b79251d06abbf38772894e74e2f/nova/objects/migrate_data.py#L195

  Because of this we don't really convert data coming from
  pre_live_migration phase and pass migrate_data object with Nones that
  are needed for setting up VNC.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1556126/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next