← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #30371

[Bug 1435633] [NEW] live migration fails at check_can_live_migrate_destination, but the status of this instance is still "migrating".

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

I have two compute node, host lxlconductor1 and host lxlcompute1.
A migration failed at function of "_call_livem_checks_on_host" because check_can_live_migrate_destination rpc call's MessagingTimeout Exception, but the status of this instance is still migrating.

2015-05-23 09:11:53.456 26651 ERROR nova.conductor.manager [req-b836290d-93d7-4bee-b974-433d067fc287 fdfd8f6a96ed4a1a9d60dbb4be1e0cf7 7f6c0898a35d4de9b53642ae984d3cf3] Migration of instance 8f50c28d-0caf-43df-b154-97f56cea7d2d to host None unexpectedly failed.
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager Traceback (most recent call last):
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 768, in _live_migrate
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     block_migration, disk_over_commit)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 201, in execute
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     return task.execute()
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 62, in execute
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     self.destination = self._find_destination()
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 173, in _find_destination
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     self._call_livem_checks_on_host(host)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 144, in _call_livem_checks_on_host
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     destination, self.block_migration, self.disk_over_commit)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 360, in check_can_live_migrate_destination
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     disk_over_commit=disk_over_commit)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     wait_for_reply=True, timeout=timeout)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     timeout=timeout)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     return self._send(target, ctxt, message, wait_for_reply, timeout)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 403, in _send
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     result = self._waiter.wait(msg_id, timeout)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 267, in wait
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     reply, ending = self._poll_connection(msg_id, timeout)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 217, in _poll_connection
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     % msg_id)
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager MessagingTimeout: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a
2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager 
2015-05-23 09:11:53.483 26651 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Migration error: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 685, in migrate_server
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     block_migration, disk_over_commit)
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 796, in _live_migrate
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     raise exception.MigrationError(reason=ex)
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher MigrationError: Migration error: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a
2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher 
2015-05-23 09:11:53.484 26651 ERROR oslo.messaging._drivers.common [-] Returning exception Migration error: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a to caller
2015-05-23 09:11:53.484 26651 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 685, in migrate_server\n    block_migration, disk_over_commit)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 796, in _live_migrate\n    raise exception.MigrationError(reason=ex)\n', 'MigrationError: Migration error: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a\n']

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1435633

Title:
   live migration fails at check_can_live_migrate_destination, but the
  status of this instance is still "migrating".

Status in OpenStack Compute (Nova):
  New

Bug description:
  I have two compute node, host lxlconductor1 and host lxlcompute1.
  A migration failed at function of "_call_livem_checks_on_host" because check_can_live_migrate_destination rpc call's MessagingTimeout Exception, but the status of this instance is still migrating.

  2015-05-23 09:11:53.456 26651 ERROR nova.conductor.manager [req-b836290d-93d7-4bee-b974-433d067fc287 fdfd8f6a96ed4a1a9d60dbb4be1e0cf7 7f6c0898a35d4de9b53642ae984d3cf3] Migration of instance 8f50c28d-0caf-43df-b154-97f56cea7d2d to host None unexpectedly failed.
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager Traceback (most recent call last):
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 768, in _live_migrate
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     block_migration, disk_over_commit)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 201, in execute
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     return task.execute()
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 62, in execute
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     self.destination = self._find_destination()
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 173, in _find_destination
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     self._call_livem_checks_on_host(host)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/conductor/tasks/live_migrate.py", line 144, in _call_livem_checks_on_host
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     destination, self.block_migration, self.disk_over_commit)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/nova/compute/rpcapi.py", line 360, in check_can_live_migrate_destination
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     disk_over_commit=disk_over_commit)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     wait_for_reply=True, timeout=timeout)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     timeout=timeout)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     return self._send(target, ctxt, message, wait_for_reply, timeout)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 403, in _send
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     result = self._waiter.wait(msg_id, timeout)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 267, in wait
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     reply, ending = self._poll_connection(msg_id, timeout)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 217, in _poll_connection
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager     % msg_id)
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager MessagingTimeout: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a
  2015-05-23 09:11:53.456 26651 TRACE nova.conductor.manager 
  2015-05-23 09:11:53.483 26651 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: Migration error: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 685, in migrate_server
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     block_migration, disk_over_commit)
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 796, in _live_migrate
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher     raise exception.MigrationError(reason=ex)
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher MigrationError: Migration error: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a
  2015-05-23 09:11:53.483 26651 TRACE oslo.messaging.rpc.dispatcher 
  2015-05-23 09:11:53.484 26651 ERROR oslo.messaging._drivers.common [-] Returning exception Migration error: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a to caller
  2015-05-23 09:11:53.484 26651 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 685, in migrate_server\n    block_migration, disk_over_commit)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 796, in _live_migrate\n    raise exception.MigrationError(reason=ex)\n', 'MigrationError: Migration error: Timed out waiting for a reply to message ID ff504056d4cd462a9fca96d3cfa8183a\n']

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1435633/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next