yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1379581] [NEW] live migration (block migration) fails at post_live_migration_at_destination function, but the status of this instance is still "migrating",

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rong Han ZTE <spring068@xxxxxxx>
Date: Fri, 10 Oct 2014 00:46:45 -0000
Reply-to: Bug 1379581 <1379581@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I have two compute node, host opencos114-93 and host opencos179-24.
A migration failed at function of " post_live_migration_at_destination", but the status of this instance is still migrating.

the log is as follow:
2014-10-09 19:25:00.678 32360 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Unexpected vif_type=binding_failed
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 401, in decorated_function
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     payload)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 311, in decorated_function
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298, in decorated_function
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4740, in post_live_migration_at_destination
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     block_migration, block_device_info)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4901, in post_live_migration_at_destination
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     write_to_disk=True)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3602, in to_xml
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     disk_info, rescue, block_device_info)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3400, in get_guest_config
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     flavor)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 481, in get_config
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     _("Unexpected vif_type=%s") % vif_type)
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher NovaException: Unexpected vif_type=binding_failed
2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher 
2014-10-09 19:25:00.679 32360 ERROR oslo.messaging._drivers.common [-] Returning exception Unexpected vif_type=binding_failed to caller
2014-10-09 19:25:00.679 32360 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 401, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped\n    payload)\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 311, in decorated_function\n    e, sys.exc_info())\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4740, in post_live_migration_at_destination\n    block_migration, block_device_info)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4901, in post_live_migration_at_destination\n    write_to_disk=True)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3602, in to_xml\n    disk_info, rescue, block_device_info)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3400, in get_guest_config\n    flavor)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 481, in get_config\n    _("Unexpected vif_type=%s") % vif_type)\n', 'NovaException: Unexpected vif_type=binding_failed\n']
2014-10-09 19:25:39.596 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Auditing locally available compute resources
2014-10-09 19:25:39.811 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free ram (MB): 1342
2014-10-09 19:25:39.811 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free disk (GB): 47
2014-10-09 19:25:39.812 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free VCPUS: 2
2014-10-09 19:25:39.855 32360 INFO nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Compute_service record updated for opencos114-93:opencos114-93
2014-10-09 19:25:39.919 32360 WARNING nova.compute.manager [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Found 0 in the database and 1 on the hypervisor.
2014-10-09 19:26:39.709 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Auditing locally available compute resources
2014-10-09 19:26:39.897 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free ram (MB): 1342
2014-10-09 19:26:39.897 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free disk (GB): 47
2014-10-09 19:26:39.897 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free VCPUS: 2
2014-10-09 19:26:39.940 32360 INFO nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Compute_service record updated for opencos114-93:opencos114-93
2014-10-09 19:26:40.001 32360 WARNING nova.compute.manager [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Found 0 in the database and 1 on the hypervisor.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1379581

Title:
  live migration (block migration) fails at
  post_live_migration_at_destination function, but the status of this
  instance is still "migrating",

Status in OpenStack Compute (Nova):
  New

Bug description:
  I have two compute node, host opencos114-93 and host opencos179-24.
  A migration failed at function of " post_live_migration_at_destination", but the status of this instance is still migrating.

  the log is as follow:
  2014-10-09 19:25:00.678 32360 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Unexpected vif_type=binding_failed
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 401, in decorated_function
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     payload)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 311, in decorated_function
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298, in decorated_function
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4740, in post_live_migration_at_destination
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     block_migration, block_device_info)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4901, in post_live_migration_at_destination
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     write_to_disk=True)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3602, in to_xml
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     disk_info, rescue, block_device_info)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3400, in get_guest_config
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     flavor)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 481, in get_config
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher     _("Unexpected vif_type=%s") % vif_type)
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher NovaException: Unexpected vif_type=binding_failed
  2014-10-09 19:25:00.678 32360 TRACE oslo.messaging.rpc.dispatcher 
  2014-10-09 19:25:00.679 32360 ERROR oslo.messaging._drivers.common [-] Returning exception Unexpected vif_type=binding_failed to caller
  2014-10-09 19:25:00.679 32360 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 401, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped\n    payload)\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 311, in decorated_function\n    e, sys.exc_info())\n', '  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 298, in decorated_function\n    return function(self, context, *args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4740, in post_live_migration_at_destination\n    block_migration, block_device_info)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4901, in post_live_migration_at_destination\n    write_to_disk=True)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3602, in to_xml\n    disk_info, rescue, block_device_info)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3400, in get_guest_config\n    flavor)\n', '  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 481, in get_config\n    _("Unexpected vif_type=%s") % vif_type)\n', 'NovaException: Unexpected vif_type=binding_failed\n']
  2014-10-09 19:25:39.596 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Auditing locally available compute resources
  2014-10-09 19:25:39.811 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free ram (MB): 1342
  2014-10-09 19:25:39.811 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free disk (GB): 47
  2014-10-09 19:25:39.812 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free VCPUS: 2
  2014-10-09 19:25:39.855 32360 INFO nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Compute_service record updated for opencos114-93:opencos114-93
  2014-10-09 19:25:39.919 32360 WARNING nova.compute.manager [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Found 0 in the database and 1 on the hypervisor.
  2014-10-09 19:26:39.709 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Auditing locally available compute resources
  2014-10-09 19:26:39.897 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free ram (MB): 1342
  2014-10-09 19:26:39.897 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free disk (GB): 47
  2014-10-09 19:26:39.897 32360 AUDIT nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Free VCPUS: 2
  2014-10-09 19:26:39.940 32360 INFO nova.compute.resource_tracker [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Compute_service record updated for opencos114-93:opencos114-93
  2014-10-09 19:26:40.001 32360 WARNING nova.compute.manager [req-80344faa-4327-4a56-8bef-7af77e23cb32 None None] Found 0 in the database and 1 on the hypervisor.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1379581/+subscriptions
Follow ups

[Bug 1379581] Re: live migration (block migration) fails at post_live_migration_at_destination function, but the status of this instance is still "migrating".
From: OpenStack Infra, 2016-02-18
[Bug 1379581] [NEW] live migration (block migration) fails at post_live_migration_at_destination function, but the status of this instance is still "migrating",
From: Rong Han ZTE, 2014-10-10
References

[Bug 1379581] [NEW] live migration (block migration) fails at post_live_migration_at_destination function, but the status of this instance is still "migrating",
From: Rong Han ZTE, 2014-10-10