yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1598860] [NEW] live migration failing

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dave Walker <email@xxxxxxxxxx>
Date: Mon, 04 Jul 2016 14:09:19 -0000
Reply-to: Bug 1598860 <1598860@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Deploying OpenStack current master via kolla source, with kvm and
ceph/rbd enabled and attempting to perform a live-migration, it becomes
stuck in state migrating with this appearing in the nova-compute log.

Looking at nova git log, i'm seeing a bunch of live-migration changes in
the last few days.  I suspect there might have been a regression.

2016-07-04 10:00:15.643 1 INFO nova.compute.resource_tracker [req-1f6df1d8-a88a-4f27-8ce2-609ea25da4e2 - - - - -] Compute_service record updated for compute01:compute01
2016-07-04 10:00:56.731 1 ERROR root [req-a2b97b2c-f479-4b7c-b3d9-142c1ab1b25f b3bedc85b7674ce2b7e0589a7427dc81 22ab55c3ffe343639d872b1e4db6abb4 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 608, in decorated_function\n    *args, **kwargs)\n', '  File "/usr/lib64/python2.7/inspect.py", line 980, in getcallargs\n    \'arguments\' if num_required > 1 else \'argument\', num_total))\n', 'TypeError: live_migration() takes exactly 7 arguments (6 given)\n']
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server [req-a2b97b2c-f479-4b7c-b3d9-142c1ab1b25f b3bedc85b7674ce2b7e0589a7427dc81 22ab55c3ffe343639d872b1e4db6abb4 - - -] Exception during message handling
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 71, in wrapped
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     f, self, context, *args, **kw)
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 85, in _get_call_dict
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     context, *args, **kw)
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/inspect.py", line 980, in getcallargs
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     'arguments' if num_required > 1 else 'argument', num_total))
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server TypeError: live_migration() takes exactly 7 arguments (6 given)
2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1598860

Title:
  live migration failing

Status in OpenStack Compute (nova):
  New

Bug description:
  Deploying OpenStack current master via kolla source, with kvm and
  ceph/rbd enabled and attempting to perform a live-migration, it
  becomes stuck in state migrating with this appearing in the nova-
  compute log.

  Looking at nova git log, i'm seeing a bunch of live-migration changes
  in the last few days.  I suspect there might have been a regression.

  2016-07-04 10:00:15.643 1 INFO nova.compute.resource_tracker [req-1f6df1d8-a88a-4f27-8ce2-609ea25da4e2 - - - - -] Compute_service record updated for compute01:compute01
  2016-07-04 10:00:56.731 1 ERROR root [req-a2b97b2c-f479-4b7c-b3d9-142c1ab1b25f b3bedc85b7674ce2b7e0589a7427dc81 22ab55c3ffe343639d872b1e4db6abb4 - - -] Original exception being dropped: ['Traceback (most recent call last):\n', '  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 66, in wrapped\n    return f(self, context, *args, **kw)\n', '  File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 608, in decorated_function\n    *args, **kwargs)\n', '  File "/usr/lib64/python2.7/inspect.py", line 980, in getcallargs\n    \'arguments\' if num_required > 1 else \'argument\', num_total))\n', 'TypeError: live_migration() takes exactly 7 arguments (6 given)\n']
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server [req-a2b97b2c-f479-4b7c-b3d9-142c1ab1b25f b3bedc85b7674ce2b7e0589a7427dc81 22ab55c3ffe343639d872b1e4db6abb4 - - -] Exception during message handling
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 71, in wrapped
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     f, self, context, *args, **kw)
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 85, in _get_call_dict
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     context, *args, **kw)
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/inspect.py", line 980, in getcallargs
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server     'arguments' if num_required > 1 else 'argument', num_total))
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server TypeError: live_migration() takes exactly 7 arguments (6 given)
  2016-07-04 10:00:56.732 1 ERROR oslo_messaging.rpc.server

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1598860/+subscriptions
Follow ups

[Bug 1598860] Re: live migration failing when nova-conductor is started before other services
From: OpenStack Infra, 2016-08-02