yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #26212
[Bug 1377644] Re: Instance stuck at migrating status when live migrate between icehouse node and juno node
** Changed in: nova
Status: Fix Committed => Fix Released
** Changed in: nova
Milestone: None => kilo-1
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1377644
Title:
Instance stuck at migrating status when live migrate between icehouse
node and juno node
Status in OpenStack Compute (Nova):
Fix Released
Bug description:
When live migrate instance between icehouse node to juno node, the nova client is stuck at call:
REQ: curl -i 'http://cloudcontroller:8774/v2/e4c500071fe448989c86563b7c685a47/servers/be356982-71d3-46db-9964-a72f90cfed2d/action' -X POST -H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: python-novaclient" -H "X-Auth-Project-Id: demo" -H "X-Auth-Token: {SHA1}3ad85671637090f616bd6b4fb78d0847b5491ce8" -d '{"os-migrateLive": {"disk_over_commit": false, "block_migration": true, "host": "os3"}}'
And the instance status stuck at migrating.
And get error message from nova-conductor:
2014-10-05 22:20:20.461 ERROR nova.conductor.manager [req-b80cccfc-be96-471a-90dc-ade51e09a86f admin demo] Migration of instance be356982-71d3-46db-9964-a72f90
cfed2d to host os3 unexpectedly failed.
2014-10-05 22:20:20.461 TRACE nova.conductor.manager Traceback (most recent call last):
2014-10-05 22:20:20.461 TRACE nova.conductor.manager File "/opt/stack/nova/nova/conductor/manager.py", line 562, in _live_migrate
2014-10-05 22:20:20.461 TRACE nova.conductor.manager block_migration, disk_over_commit)
2014-10-05 22:20:20.461 TRACE nova.conductor.manager File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 191, in execute
2014-10-05 22:20:20.461 TRACE nova.conductor.manager return task.execute()
2014-10-05 22:20:20.461 TRACE nova.conductor.manager File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 61, in execute
2014-10-05 22:20:20.461 TRACE nova.conductor.manager self._check_requested_destination()
2014-10-05 22:20:20.461 TRACE nova.conductor.manager File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 99, in _check_requested_destination
2014-10-05 22:20:20.461 TRACE nova.conductor.manager self._call_livem_checks_on_host(self.destination)
2014-10-05 22:20:20.461 TRACE nova.conductor.manager File "/opt/stack/nova/nova/conductor/tasks/live_migrate.py", line 141, in _call_livem_checks_on_host
2014-10-05 22:20:20.461 TRACE nova.conductor.manager destination, self.block_migration, self.disk_over_commit)
2014-10-05 22:20:20.461 TRACE nova.conductor.manager File "/opt/stack/nova/nova/compute/rpcapi.py", line 349, in check_can_live_migrate_destination
2014-10-05 22:20:20.461 TRACE nova.conductor.manager self._check_live_migration_api_version(destination)
2014-10-05 22:20:20.461 TRACE nova.conductor.manager File "/opt/stack/nova/nova/compute/rpcapi.py", line 296, in _check_live_migration_api_version
2014-10-05 22:20:20.461 TRACE nova.conductor.manager raise exception.LiveMigrationWithOldNovaNotSafe(server=server)
2014-10-05 22:20:20.461 TRACE nova.conductor.manager LiveMigrationWithOldNovaNotSafe: Host os3 is running an old version of Nova, live migrations involving tha
t version may cause data loss. Upgrade Nova on os3 and try again.
2014-10-05 22:20:20.461 TRACE nova.conductor.manager
2014-10-05 22:20:20.462 ERROR oslo.messaging.rpc.dispatcher [req-b80cccfc-be96-471a-90dc-ade51e09a86f admin demo] Exception during message handling: Migration error: Host os3 is running an old version of Nova, live migrations involving that version may cause data loss. Upgrade Nova on os3 and try again.
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher return func(*args, **kwargs)
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/conductor/manager.py", line 482, in migrate_server
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher block_migration, disk_over_commit)
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/conductor/manager.py", line 590, in _live_migrate
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher raise exception.MigrationError(reason=ex)
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher MigrationError: Migration error: Host os3 is running an old version of Nova, live migrations involving that version may cause data loss. Upgrade Nova on os3 and try again.
2014-10-05 22:20:20.462 TRACE oslo.messaging.rpc.dispatcher
2014-10-05 22:20:20.463 ERROR oslo.messaging._drivers.common [req-b80cccfc-be96-471a-90dc-ade51e09a86f admin demo] Returning exception Migration error: Host os3 is running an old version of Nova, live migrations involving that version may cause data loss. Upgrade Nova on os3 and try again. to caller
2014-10-05 22:20:20.463 ERROR oslo.messaging._drivers.common [req-b80cccfc-be96-471a-90dc-ade51e09a86f admin demo] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/server.py", line 139, in inner\n return func(*args, **kwargs)\n', ' File "/opt/stack/nova/nova/conductor/manager.py", line 482, in migrate_server\n block_migration, disk_over_commit)\n', ' File "/opt/stack/nova/nova/conductor/manager.py", line 590, in _live_migrate\n raise exception.MigrationError(reason=ex)\n', 'MigrationError: Migration error: Host os3 is running an old version of Nova, live migrations involving that version may cause data loss. Upgrade Nova on os3 and try again.\n']
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
listener.cb(fileno)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 215, in main
self._resolve_links()
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 230, in _resolve_links
f(self, *ca, **ckw)
File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_executors/impl_eventlet.py", line 46, in complete
thread.wait()
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 173, in wait
return self._exit_event.wait()
File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 125, in wait
current.throw(*self._exc)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
result = function(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 129, in <lambda>
yield lambda: self._dispatch_and_reply(incoming)
File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 144, in _dispatch_and_reply
incoming.reply(failure=exc_info)
File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 72, in reply
self._send_reply(conn, reply, failure, log_failure=log_failure)
File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 49, in _send_reply
log_failure)
File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 217, in serialize_remote_exception
json_data = jsonutils.dumps(data)
File "/usr/local/lib/python2.7/dist-packages/oslo/serialization/jsonutils.py", line 186, in dumps
return json.dumps(obj, default=default, **kwargs)
File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
sort_keys=sort_keys, **kw).encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
ValueError: Circular reference detected
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1377644/+subscriptions
References