yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1412279] Re: live migration, nova-conductor raise a exception named 'MigrationError', but nova-api can not receive any error reply except a 'MessageTimeOut' exception occurs.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Mon, 09 Feb 2015 09:24:21 -0000
Reply-to: Bug 1412279 <1412279@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
*** This bug is a duplicate of bug 1397153 ***
    https://bugs.launchpad.net/bugs/1397153

** This bug has been marked a duplicate of bug 1397153
   instance stuck at migrating state when some unexpected error happened

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1412279

Title:
  live migration, nova-conductor raise a exception named
  'MigrationError', but nova-api can not receive any error reply except
  a 'MessageTimeOut' exception occurs.

Status in OpenStack Compute (Nova):
  Fix Committed

Bug description:
  live migration,  nova-api wait for a reply of rpc_call to nova-
  conductor, nova-conductor raise a exception named 'MigrationError'.
  The nova-api service can not receive any error reply from nova-
  conductor service, and  a 'MessageTimeOut' exception occurs until 60
  seconds passed.

  nova-api.log
  2015-03-20 09:23:39.066 20276 ERROR nova.api.openstack.compute.contrib.admin_actions [req-3d2de7b2-5258-4ba0-935d-48eb84c877fc 7aba40236a4c4494aba4eb0b9365ffee be568b8239d147e58f1ef16b6011c93d] Live migration of instance ae701ba0-1eed-4d0f-a0c3-052b1f433553 to another host failed
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions Traceback (most recent call last):
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/admin_actions.py", line 338, in _migrate_live
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     disk_over_commit, host)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 238, in _wrapped
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     return fn(self, context, instance, *args, **kwargs)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 192, in inner
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     return f(self, context, instance, *args, **kw)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3233, in live_migrate
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     disk_over_commit=disk_over_commit)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/nova/conductor/api.py", line 387, in live_migrate_instance
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     block_migration, disk_over_commit, None)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/nova/conductor/rpcapi.py", line 476, in migrate_server
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     reservations=reservations)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     wait_for_reply=True, timeout=timeout)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     timeout=timeout)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     return self._send(target, ctxt, message, wait_for_reply, timeout)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 403, in _send
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     result = self._waiter.wait(msg_id, timeout)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 267, in wait
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     reply, ending = self._poll_connection(msg_id, timeout)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 217, in _poll_connection
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions     % msg_id)
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions MessagingTimeout: Timed out waiting for a reply to message ID 20ee328ddd9446bda761fc162fb00d50
  2015-03-20 09:23:39.066 20276 TRACE nova.api.openstack.compute.contrib.admin_actions 

  
  nova-conductor.log
  015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 685, in migrate_server
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     block_migration, disk_over_commit)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 793, in _live_migrate
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     raise exception.MigrationError(reason=ex)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher MigrationError: Migration error: [Errno [Errno 2] No such file or directory: '/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config'
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     payload)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 373, in decorated_function
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4684, in check_can_live_migrate_source
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     dest_check_data)
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4610, in check_can_live_migrate_source
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     jsonutils.loads(self.get_instance_disk_info(instance['name'])))
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5293, in get_instance_disk_info
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     dk_size = int(os.path.getsize(path))
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib64/python2.7/genericpath.py", line 49, in getsize
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher     return os.stat(filename).st_size
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config'
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher ] No such file or directory
  2015-03-20 09:22:39.742 13849 TRACE oslo.messaging.rpc.dispatcher 
  2015-03-20 09:22:39.743 13849 ERROR oslo.messaging._drivers.common [-] Returning exception Migration error: [Errno [Errno 2] No such file or directory: '/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config'
  Traceback (most recent call last):

    File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
      incoming.message))

    File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
      return self._do_dispatch(endpoint, method, ctxt, args)

    File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
      result = getattr(endpoint, method)(ctxt, **new_args)

    File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
      payload)

    File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
      return f(self, context, *args, **kw)

    File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 373, in decorated_function
      e, sys.exc_info())

    File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
      six.reraise(self.type_, self.value, self.tb)

    File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function
      return function(self, context, *args, **kwargs)

    File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4684, in check_can_live_migrate_source
      dest_check_data)

    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4610, in check_can_live_migrate_source
      jsonutils.loads(self.get_instance_disk_info(instance['name'])))

    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5293, in get_instance_disk_info
      dk_size = int(os.path.getsize(path))

    File "/usr/lib64/python2.7/genericpath.py", line 49, in getsize
      return os.stat(filename).st_size

  OSError: [Errno 2] No such file or directory: '/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config'
  ] No such file or directory to caller
  2015-03-20 09:22:39.743 13849 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\n    return func(*args, **kwargs)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 685, in migrate_server\n    block_migration, disk_over_commit)\n', '  File "/usr/lib/python2.7/site-packages/nova/conductor/manager.py", line 793, in _live_migrate\n    raise exception.MigrationError(reason=ex)\n', 'MigrationError: Migration error: [Errno [Errno 2] No such file or directory: \'/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config\'\nTraceback (most recent call last):\n\n  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n\n  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n\n  File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n\n  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped\n    payload)\n\n  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n\n  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped\n    return f(self, context, *args, **kw)\n\n  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 373, in decorated_function\n    e, sys.exc_info())\n\n  File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n    six.reraise(self.type_, self.value, self.tb)\n\n  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function\n    return function(self, context, *args, **kwargs)\n\n  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4684, in check_can_live_migrate_source\n    dest_check_data)\n\n  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4610, in check_can_live_migrate_source\n    jsonutils.loads(self.get_instance_disk_info(instance[\'name\'])))\n\n  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5293, in get_instance_disk_info\n    dk_size = int(os.path.getsize(path))\n\n  File "/usr/lib64/python2.7/genericpath.py", line 49, in getsize\n    return os.stat(filename).st_size\n\nOSError: [Errno 2] No such file or directory: \'/var/lib/nova/instances/ae701ba0-1eed-4d0f-a0c3-052b1f433553/disk.config\'\n] No such file or directory\n']

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1412279/+subscriptions
References

[Bug 1412279] [NEW] live migration, nova-conductor raise a exception named 'MigrationError', but nova-api can not receive any error reply except a 'MessageTimeOut' exception occurs.
From: Rong Han ZTE, 2015-01-19