← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #31120

[Bug 1441054] [NEW] live-migration --block-migrate fails with default libvirt flags

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

while trying to live-migrate an instance with the --block-migrate
option, I've got an error on the host which hosts the VM :


2015-04-07 11:01:32.554 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Starting monitoring of live migration from (pid=5202) _live_migration /opt/stack/nova/nova/virt/libvirt/driver.py:5642
2015-04-07 11:01:32.556 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Operation thread is still running from (pid=5202) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5494
2015-04-07 11:01:32.557 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Migration not running yet from (pid=5202) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5525
2015-04-07 11:01:33.142 INFO nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Migration running for 0 secs, memory 0% remaining; (bytes processed=0, remaining=0, total=0)
2015-04-07 11:01:33.277 ERROR nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Live Migration failure: End of file while reading data: Input/output error
2015-04-07 11:01:33.278 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Migration operation thread notification from (pid=5202) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:5633
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
    waiter.switch(result)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5428, in _live_migration_operation
    instance=instance)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5397, in _live_migration_operation
    CONF.libvirt.live_migration_bandwidth)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
    six.reraise(c, e, tb)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1734, in migrateToURI2
    if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: End of file while reading data: Input/output error
2015-04-07 11:01:33.644 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] VM running on src, migration failed from (pid=5202) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5500
2015-04-07 11:01:33.645 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Fixed incorrect job type to be 4 from (pid=5202) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5520
2015-04-07 11:01:33.645 ERROR nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Migration operation has aborted
2015-04-07 11:01:33.733 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Live migration monitoring is all done from (pid=5202) _live_migration /opt/stack/nova/nova/virt/libvirt/driver.py:5653


live migration with block-migrate works fine when I remove the flag VIR_MIGRATE_TUNNELLED from the option block_migration_flag.

indeed, live block migration cannot occur in tunneling mode, as reported here : 
https://wiki.openstack.org/wiki/OSSN/OSSN-0007

** Affects: nova
     Importance: Undecided
     Assignee: Mathieu Rohon (mathieu-rohon)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1441054

Title:
  live-migration --block-migrate fails with default libvirt flags

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  while trying to live-migrate an instance with the --block-migrate
  option, I've got an error on the host which hosts the VM :

  
  2015-04-07 11:01:32.554 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Starting monitoring of live migration from (pid=5202) _live_migration /opt/stack/nova/nova/virt/libvirt/driver.py:5642
  2015-04-07 11:01:32.556 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Operation thread is still running from (pid=5202) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5494
  2015-04-07 11:01:32.557 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Migration not running yet from (pid=5202) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5525
  2015-04-07 11:01:33.142 INFO nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Migration running for 0 secs, memory 0% remaining; (bytes processed=0, remaining=0, total=0)
  2015-04-07 11:01:33.277 ERROR nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Live Migration failure: End of file while reading data: Input/output error
  2015-04-07 11:01:33.278 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Migration operation thread notification from (pid=5202) thread_finished /opt/stack/nova/nova/virt/libvirt/driver.py:5633
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
      timer()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
      cb(*args, **kw)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
      waiter.switch(result)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5428, in _live_migration_operation
      instance=instance)
    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5397, in _live_migration_operation
      CONF.libvirt.live_migration_bandwidth)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
      six.reraise(c, e, tb)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1734, in migrateToURI2
      if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
  libvirtError: End of file while reading data: Input/output error
  2015-04-07 11:01:33.644 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] VM running on src, migration failed from (pid=5202) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5500
  2015-04-07 11:01:33.645 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Fixed incorrect job type to be 4 from (pid=5202) _live_migration_monitor /opt/stack/nova/nova/virt/libvirt/driver.py:5520
  2015-04-07 11:01:33.645 ERROR nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Migration operation has aborted
  2015-04-07 11:01:33.733 DEBUG nova.virt.libvirt.driver [-] [instance: 31b63d63-b392-4197-8864-b6d85dae438f] Live migration monitoring is all done from (pid=5202) _live_migration /opt/stack/nova/nova/virt/libvirt/driver.py:5653

  
  live migration with block-migrate works fine when I remove the flag VIR_MIGRATE_TUNNELLED from the option block_migration_flag.

  indeed, live block migration cannot occur in tunneling mode, as reported here : 
  https://wiki.openstack.org/wiki/OSSN/OSSN-0007

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1441054/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next