yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1377644] Re: Instance stuck at migrating status when live migrate between icehouse node and juno node

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Thu, 18 Dec 2014 20:03:03 -0000
Reply-to: Bug 1377644 <1377644@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => kilo-1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1377644

Title:
  Instance stuck at migrating status when live migrate between icehouse
  node and juno node

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  When live migrate instance between icehouse node to juno node, the nova client is stuck at call:
  REQ: curl -i 'http://cloudcontroller:8774/v2/e4c500071fe448989c86563b7c685a47/servers/be356982-71d3-46db-9964-a72f90cfed2d/action' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: demo" -H "X-Auth-Token: {SHA1}3ad85671637090f616bd6b4fb78d0847b5491ce8" -d '{"os-migrateLive": {"disk_over_commit": false, "block_migration": true, "host": "os3"}}'

  
  And the instance status stuck at migrating.

  
  And get error message from nova-conductor:
  2014-10-05 22:20:20.461 ERROR nova.conductor.manager [req-b80cccfc-be96-471a-90dc-ade51e09a86f admin demo] Migration of instance be356982-71d3-46db-9964-a72f90
  cfed2d to host os3 unexpectedly failed.
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager Traceback (most recent call last):
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager   File "/opt/stack/nova/nova/conductor/manager.py", line 562, in _live_migrate
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager     block_migration, disk_over_commit)
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager   File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 191, in execute
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager     return task.execute()
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager   File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 61, in execute
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager     self._check_requested_destination()
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager   File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 99, in _check_requested_destination
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager     self._call_livem_checks_on_host(self.destination)
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager   File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 141, in _call_livem_checks_on_host
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager     destination, self.block_migration, self.disk_over_commit)
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager   File "/opt/stack/nova/nova/compute/rpcapi.py", line 349, in check_can_live_migrate_destination
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager     self._check_live_migration_api_version(destination)
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager   File "/opt/stack/nova/nova/compute/rpcapi.py", line 296, in _check_live_migration_api_version
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager     raise exception.LiveMigrationWithOldNovaNotSafe(server=server)
  2014-10-05 22:20:20.461 TRACE nova.conductor.manager LiveMigrationWithOldNovaNotSafe: Host os3 is running an old version of Nova, live migrations involving tha
  t version may cause data loss. Upgrade Nova on os3 and try again.

  2014-10-05 22:20:20.461 TRACE nova.conductor.manager 
  2014-10-05 22:20:20.462 ERROR oslo.messaging.rpc.dispatcher [req-b80cccfc-be96-471a-90dc-ade51e09a86f admin demo] Exception during message handling: Migration error: Host os3 is running an old version of Nova, live migrations involving that version may cause data loss. Upgrade Nova on os3 and try again.
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/manager.py", line 482, in migrate_server
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher     block_migration, disk_over_commit)
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher   File "/opt/stack/nova/nova/conductor/manager.py", line 590, in _live_migrate
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher     raise exception.MigrationError(reason=ex)
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher MigrationError: Migration error: Host os3 is running an old version of Nova, live migrations involving that version may cause data loss. Upgrade Nova on os3 and try again.
  2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher 
  2014-10-05 22:20:20.463 ERROR oslo.messaging._drivers.common [req-b80cccfc-be96-471a-90dc-ade51e09a86f admin demo] Returning exception Migration error: Host os3 is running an old version of Nova, live migrations involving that version may cause data loss. Upgrade Nova on os3 and try again. to caller
  2014-10-05 22:20:20.463 ERROR oslo.messaging._drivers.common [req-b80cccfc-be96-471a-90dc-ade51e09a86f admin demo] ['Traceback (most recent call last):\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner\n    return func(*args, **kwargs)\n', '  File "/opt/stack/nova/nova/conductor/manager.py", line 482, in migrate_server\n    block_migration, disk_over_commit)\n', '  File "/opt/stack/nova/nova/conductor/manager.py", line 590, in _live_migrate\n    raise exception.MigrationError(reason=ex)\n', 'MigrationError: Migration error: Host os3 is running an old version of Nova, live migrations involving that version may cause data loss. Upgrade Nova on os3 and try again.\n']

  
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
      listener.cb(fileno)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 215, in main
      self._resolve_links()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 230, in _resolve_links
      f(self, *ca, **ckw)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_executors/impl_eventlet.py", line 46, in complete
      thread.wait()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 173, in wait
      return self._exit_event.wait()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
      current.throw(*self._exc)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
      result = function(*args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 129, in <lambda>
      yield lambda: self._dispatch_and_reply(incoming)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 144, in _dispatch_and_reply
      incoming.reply(failure=exc_info)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 72, in reply
      self._send_reply(conn, reply, failure, log_failure=log_failure)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 49, in _send_reply
      log_failure)
    File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 217, in serialize_remote_exception
      json_data = jsonutils.dumps(data)
    File "/usr/local/lib/python2.7/dist-packages/oslo/serialization/jsonutils.py", line 186, in dumps
      return json.dumps(obj, default=default, **kwargs)
    File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
      sort_keys=sort_keys, **kw).encode(obj)
    File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
      chunks = self.iterencode(o, _one_shot=True)
    File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
      return _iterencode(o, 0)
  ValueError: Circular reference detected

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1377644/+subscriptions
References

[Bug 1377644] [NEW] Instance stuck at migrating status when live migrate between icehouse node and juno node
From: Alex Xu, 2014-10-05