← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #55571

[Bug 1617073] [NEW] Libvirt live block migration fails due to ['migrate-incoming': Failed to bind socket: Address already in use]

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

On the gate job gate-tempest-dsvm-multinode-live-migration, a Tempest
test test_live_block_migration fails because an actual host which a
server stays was different from the expected.

The tempest log is http://logs.openstack.org/50/349450/5/check/gate-
tempest-dsvm-multinode-live-
migration/60eb3b3/console.html#_2016-08-25_19_14_13_416233

2016-08-25 19:14:13.416233 | 2016-08-25 19:14:13.415 | Captured traceback:
2016-08-25 19:14:13.417870 | 2016-08-25 19:14:13.417 | ~~~~~~~~~~~~~~~~~~~
2016-08-25 19:14:13.419367 | 2016-08-25 19:14:13.419 |     Traceback (most recent call last):
2016-08-25 19:14:13.421015 | 2016-08-25 19:14:13.420 |       File "tempest/api/compute/admin/test_live_migration.py", line 132, in test_live_block_migration
2016-08-25 19:14:13.422730 | 2016-08-25 19:14:13.422 |         self._test_live_migration()
2016-08-25 19:14:13.424314 | 2016-08-25 19:14:13.423 |       File "tempest/api/compute/admin/test_live_migration.py", line 128, in _test_live_migration
2016-08-25 19:14:13.426047 | 2016-08-25 19:14:13.425 |         msg)
2016-08-25 19:14:13.427470 | 2016-08-25 19:14:13.427 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
2016-08-25 19:14:13.429269 | 2016-08-25 19:14:13.428 |         self.assertThat(observed, matcher, message)
2016-08-25 19:14:13.430954 | 2016-08-25 19:14:13.430 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
2016-08-25 19:14:13.432623 | 2016-08-25 19:14:13.432 |         raise mismatch_error
2016-08-25 19:14:13.434426 | 2016-08-25 19:14:13.433 |     testtools.matchers._impl.MismatchError: !=:
2016-08-25 19:14:13.436041 | 2016-08-25 19:14:13.435 |     reference = u'ubuntu-xenial-2-node-rax-ord-3874987'
2016-08-25 19:14:13.437717 | 2016-08-25 19:14:13.437 |     actual    = u'ubuntu-xenial-2-node-rax-ord-3874987-181073'
2016-08-25 19:14:13.439228 | 2016-08-25 19:14:13.438 |     : Live Migration failed. Migrations list for Instance 0240bf1c-d2cf-407f-9ecf-7acfd891b4c0: [
2016-08-25 19:14:13.440771 | 2016-08-25 19:14:13.440 |     {u'migration_type': u'live-migration', u'instance_uuid': u'0240bf1c-d2cf-407f-9ecf-7acfd891b4c0', u'status': u'error', u'updated_at': u'2016-08-25T19:13:40.000000', u'created_at': u'2016-08-25T19:13:33.000000', u'source_node': None, u'dest_node': None, u'dest_host': None, u'dest_compute': u'ubuntu-xenial-2-node-rax-ord-3874987', u'source_compute': u'ubuntu-xenial-2-node-rax-ord-3874987-181073', u'id': 1, u'old_instance_type_id': 11, u'new_instance_type_id': 11}]

nova-cpu outputs the following log:

http://logs.openstack.org/50/349450/5/check/gate-tempest-dsvm-multinode-
live-
migration/60eb3b3/logs/subnode-2/screen-n-cpu.txt.gz#_2016-08-25_19_13_38_903

2016-08-25 19:13:38.904 25185 DEBUG nova.virt.libvirt.driver [req-04d0773e-0085-4605-ad4e-bd98329b6c4d tempest-LiveAutoBlockMigrationV225TestJSON-2087766141 tempest-LiveAutoBlockMigrationV225TestJSON-2087766141] [instance: 0240bf1c-d2cf-407f-9ecf-7acfd891b4c0] Migration operation thread notification thread_finished /opt/stack/new/nova/nova/virt/libvirt/driver.py:6273
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
    waiter.switch(result)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/new/nova/nova/utils.py", line 1066, in context_wrapper
    return func(*args, **kwargs)
  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5874, in _live_migration_operation
    instance=instance)
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
    self.force_reraise()
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5870, in _live_migration_operation
    bandwidth=CONF.libvirt.live_migration_bandwidth)
  File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 568, in migrate
    destination, params=params, flags=flags)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
    rv = execute(f, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
    six.reraise(c, e, tb)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
    rv = meth(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1833, in migrateToURI3
    if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
libvirtError: internal error: unable to execute QEMU command 'migrate-incoming': Failed to bind socket: Address already in use

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1617073

Title:
  Libvirt live block migration fails due to ['migrate-incoming': Failed
  to bind socket: Address already in use]

Status in OpenStack Compute (nova):
  New

Bug description:
  On the gate job gate-tempest-dsvm-multinode-live-migration, a Tempest
  test test_live_block_migration fails because an actual host which a
  server stays was different from the expected.

  The tempest log is http://logs.openstack.org/50/349450/5/check/gate-
  tempest-dsvm-multinode-live-
  migration/60eb3b3/console.html#_2016-08-25_19_14_13_416233

  2016-08-25 19:14:13.416233 | 2016-08-25 19:14:13.415 | Captured traceback:
  2016-08-25 19:14:13.417870 | 2016-08-25 19:14:13.417 | ~~~~~~~~~~~~~~~~~~~
  2016-08-25 19:14:13.419367 | 2016-08-25 19:14:13.419 |     Traceback (most recent call last):
  2016-08-25 19:14:13.421015 | 2016-08-25 19:14:13.420 |       File "tempest/api/compute/admin/test_live_migration.py", line 132, in test_live_block_migration
  2016-08-25 19:14:13.422730 | 2016-08-25 19:14:13.422 |         self._test_live_migration()
  2016-08-25 19:14:13.424314 | 2016-08-25 19:14:13.423 |       File "tempest/api/compute/admin/test_live_migration.py", line 128, in _test_live_migration
  2016-08-25 19:14:13.426047 | 2016-08-25 19:14:13.425 |         msg)
  2016-08-25 19:14:13.427470 | 2016-08-25 19:14:13.427 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual
  2016-08-25 19:14:13.429269 | 2016-08-25 19:14:13.428 |         self.assertThat(observed, matcher, message)
  2016-08-25 19:14:13.430954 | 2016-08-25 19:14:13.430 |       File "/opt/stack/new/tempest/.tox/tempest/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
  2016-08-25 19:14:13.432623 | 2016-08-25 19:14:13.432 |         raise mismatch_error
  2016-08-25 19:14:13.434426 | 2016-08-25 19:14:13.433 |     testtools.matchers._impl.MismatchError: !=:
  2016-08-25 19:14:13.436041 | 2016-08-25 19:14:13.435 |     reference = u'ubuntu-xenial-2-node-rax-ord-3874987'
  2016-08-25 19:14:13.437717 | 2016-08-25 19:14:13.437 |     actual    = u'ubuntu-xenial-2-node-rax-ord-3874987-181073'
  2016-08-25 19:14:13.439228 | 2016-08-25 19:14:13.438 |     : Live Migration failed. Migrations list for Instance 0240bf1c-d2cf-407f-9ecf-7acfd891b4c0: [
  2016-08-25 19:14:13.440771 | 2016-08-25 19:14:13.440 |     {u'migration_type': u'live-migration', u'instance_uuid': u'0240bf1c-d2cf-407f-9ecf-7acfd891b4c0', u'status': u'error', u'updated_at': u'2016-08-25T19:13:40.000000', u'created_at': u'2016-08-25T19:13:33.000000', u'source_node': None, u'dest_node': None, u'dest_host': None, u'dest_compute': u'ubuntu-xenial-2-node-rax-ord-3874987', u'source_compute': u'ubuntu-xenial-2-node-rax-ord-3874987-181073', u'id': 1, u'old_instance_type_id': 11, u'new_instance_type_id': 11}]

  nova-cpu outputs the following log:

  http://logs.openstack.org/50/349450/5/check/gate-tempest-dsvm-
  multinode-live-
  migration/60eb3b3/logs/subnode-2/screen-n-cpu.txt.gz#_2016-08-25_19_13_38_903

  2016-08-25 19:13:38.904 25185 DEBUG nova.virt.libvirt.driver [req-04d0773e-0085-4605-ad4e-bd98329b6c4d tempest-LiveAutoBlockMigrationV225TestJSON-2087766141 tempest-LiveAutoBlockMigrationV225TestJSON-2087766141] [instance: 0240bf1c-d2cf-407f-9ecf-7acfd891b4c0] Migration operation thread notification thread_finished /opt/stack/new/nova/nova/virt/libvirt/driver.py:6273
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
      timer()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
      cb(*args, **kw)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 168, in _do_send
      waiter.switch(result)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
      result = function(*args, **kwargs)
    File "/opt/stack/new/nova/nova/utils.py", line 1066, in context_wrapper
      return func(*args, **kwargs)
    File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5874, in _live_migration_operation
      instance=instance)
    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
      self.force_reraise()
    File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 5870, in _live_migration_operation
      bandwidth=CONF.libvirt.live_migration_bandwidth)
    File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 568, in migrate
      destination, params=params, flags=flags)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
      six.reraise(c, e, tb)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 1833, in migrateToURI3
      if ret == -1: raise libvirtError ('virDomainMigrateToURI3() failed', dom=self)
  libvirtError: internal error: unable to execute QEMU command 'migrate-incoming': Failed to bind socket: Address already in use

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1617073/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next