yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1453675] Re: Live migration fails

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sylvain Bauza <sbauza@xxxxxxx>
Date: Wed, 13 May 2015 20:00:55 -0000
Reply-to: Bug 1453675 <1453675@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1453675

Title:
  Live migration fails

Status in OpenStack Compute (Nova):
  Invalid

Bug description:
  1: Exact Version (Latest apt-get dist-upgrade with Kilo repositories for ubuntu 14.04.02)
  ii  nova-api                            1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - API frontend
  ii  nova-cert                           1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - certificate management
  ii  nova-common                         1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - common files
  ii  nova-conductor                      1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - conductor service
  ii  nova-consoleauth                    1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - Console Authenticator
  ii  nova-novncproxy                     1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - NoVNC proxy
  ii  nova-scheduler                      1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute - virtual machine scheduler
  ii  python-nova                         1:2015.1~rc1-0ubuntu1~cloud0          all          OpenStack Compute Python libraries
  ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

  2: Log files
  2015-05-11 09:26:05.515 25372 DEBUG nova.compute.api [req-d3f3807f-3aa9-472e-b076-3987372f8943 fb2aaa72c412443fafe9d483ecb396c5 3d5adc9afa334a2097fc4374fe3c96e1 - - -] [instance: 9cf946cf-8e0a-4e4b-8651-514251f7c2de] Going to try to live migrate instance to compute2 live_migrate /usr/lib/python2.7/dist-packages/nova/compute/api.py:3224
  2015-05-11 09:26:05.607 25372 INFO oslo_messaging._drivers.impl_rabbit [req-d3f3807f-3aa9-472e-b076-3987372f8943 fb2aaa72c412443fafe9d483ecb396c5 3d5adc9afa334a2097fc4374fe3c96e1 - - -] Connecting to AMQP server on controller:5672
  2015-05-11 09:26:05.619 25372 INFO oslo_messaging._drivers.impl_rabbit [req-d3f3807f-3aa9-472e-b076-3987372f8943 fb2aaa72c412443fafe9d483ecb396c5 3d5adc9afa334a2097fc4374fe3c96e1 - - -] Connected to AMQP server on controller:5672
  2015-05-11 09:26:05.623 25372 INFO oslo_messaging._drivers.impl_rabbit [req-d3f3807f-3aa9-472e-b076-3987372f8943 fb2aaa72c412443fafe9d483ecb396c5 3d5adc9afa334a2097fc4374fe3c96e1 - - -] Connecting to AMQP server on controller:5672
  2015-05-11 09:26:05.636 25372 INFO oslo_messaging._drivers.impl_rabbit [req-d3f3807f-3aa9-472e-b076-3987372f8943 fb2aaa72c412443fafe9d483ecb396c5 3d5adc9afa334a2097fc4374fe3c96e1 - - -] Connected to AMQP server on controller:5672
  2015-05-11 09:26:05.776 25372 ERROR nova.api.openstack.compute.contrib.admin_actions [req-d3f3807f-3aa9-472e-b076-3987372f8943 fb2aaa72c412443fafe9d483ecb396c5 3d5adc9afa334a2097fc4374fe3c96e1 - - -] Live migration of instance 9cf946cf-8e0a-4e4b-8651-514251f7c2de to host compute2 failed
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions Traceback (most recent call last):
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/admin_actions.py", line 331, in _migrate_live
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     disk_over_commit, host)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 219, in inner
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     return function(self, context, instance, *args, **kwargs)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 247, in _wrapped
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     return fn(self, context, instance, *args, **kwargs)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 200, in inner
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     return f(self, context, instance, *args, **kw)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3234, in live_migrate
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     disk_over_commit=disk_over_commit)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 333, in live_migrate_instance
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     block_migration, disk_over_commit, None)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 391, in migrate_server
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     return cctxt.call(context, 'migrate_server', **kw)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     retry=self.retry)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     timeout=timeout, retry=retry)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     retry=retry)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 341, in _send
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     raise result
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions MigrationError_Remote: Migration error: check_can_live_migrate_destination() takes exactly 6 arguments (5 given)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions Traceback (most recent call last):
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     executor_callback))
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     executor_callback)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     result = func(ctxt, **new_args)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions TypeError: check_can_live_migrate_destination() takes exactly 6 arguments (5 given)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions Traceback (most recent call last):
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     executor_callback))
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     executor_callback)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     result = func(ctxt, **new_args)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 142, in inner
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     return func(*args, **kwargs)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 511, in migrate_server
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     block_migration, disk_over_commit)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 638, in _live_migrate
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     raise exception.MigrationError(reason=six.text_type(ex))
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions MigrationError: Migration error: check_can_live_migrate_destination() takes exactly 6 arguments (5 given)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions Traceback (most recent call last):
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     executor_callback))
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     executor_callback)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions     result = func(ctxt, **new_args)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions TypeError: check_can_live_migrate_destination() takes exactly 6 arguments (5 given)
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.776 25372 TRACE nova.api.openstack.compute.contrib.admin_actions
  2015-05-11 09:26:05.780 25372 INFO nova.api.openstack.wsgi [req-d3f3807f-3aa9-472e-b076-3987372f8943 fb2aaa72c412443fafe9d483ecb396c5 3d5adc9afa334a2097fc4374fe3c96e1 - - -] HTTP exception thrown: Live migration of instance 9cf946cf-8e0a-4e4b-8651-514251f7c2de to host compute2 failed
  2015-05-11 09:26:05.781 25372 DEBUG nova.api.openstack.wsgi [req-d3f3807f-3aa9-472e-b076-3987372f8943 fb2aaa72c412443fafe9d483ecb396c5 3d5adc9afa334a2097fc4374fe3c96e1 - - -] Returning 500 to user: Live migration of instance 9cf946cf-8e0a-4e4b-8651-514251f7c2de to host compute2 failed __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:1166
  2015-05-11 09:26:05.782 25372 INFO nova.osapi_compute.wsgi.server [req-d3f3807f-3aa9-472e-b076-3987372f8943 fb2aaa72c412443fafe9d483ecb396c5 3d5adc9afa334a2097fc4374fe3c96e1 - - -] 10.0.0.11 "POST /v2/3d5adc9afa334a2097fc4374fe3c96e1/servers/9cf946cf-8e0a-4e4b-8651-514251f7c2de/action HTTP/1.1" status: 500 len: 364 time: 0.3190989

  3: Reproduce Steps
   3.1) nova list
   3.2) nova service-list
   3.3) nova live-migration <instance uuid> compute2 

   Expected result: 
      Migration would give successful migration from compute1 to compute2 information

   Actual result:
      nova commmand (3.3) gives error:  ERROR (ClientException): Live migration of instance 9cf946cf-8e0a-4e4b-8651-514251f7c2de to host compute2 failed (HTTP 500) (Request-ID: req-d3f3807f-3aa9-472e-b076-3987372f8943)
      Also the instance is jammed so bad that it must be deleted and if it is using volumes, the volume keeps attachment to the instance even after the instance is deleted.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1453675/+subscriptions
References

[Bug 1453675] [NEW] Live migration fails
From: Mika Saari, 2015-05-11