yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1535346] Re: Direct Snapshot in Ceph patch causes nova live-migration to break

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Sean Dague <sean@xxxxxxxxx>
Date: Wed, 17 Feb 2016 16:32:00 -0000
Reply-to: Bug 1535346 <1535346@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Given the clearly invalid python in the stack trace that mriedem pointed
out, marking as Invalid.

** Changed in: nova
       Status: Incomplete => Invalid

** Changed in: nova
   Importance: High => Low

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535346

Title:
  Direct Snapshot in Ceph patch causes nova live-migration to break

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Greetings,

  When trying to use nova live-migration we do have this trace on the compute node logs (nova-compute.log)
  The Vm stays in "MIGRATING" state, the VM is still up on the originating node, and never moves.

  VERSION IN USE: ( KILO v1 )

  root@compute2:~# dpkg -l | grep nova
  ii  nova-common                         1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - common files
  ii  nova-compute                        1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                    1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node libvirt support
  ii  python-nova                         1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
  ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

  CMD:

  nova migrate-live 6b91ffb3-bf84-48e9-b07d-99603c89055c compute4

  LOGS (/var/log/nova/nova-compute.log):

  2016-01-18 09:55:46.160 18571 ERROR oslo_messaging.rpc.dispatcher [req-fae37d19-0867-4796-9df3-b08be2120d15 ad8ac2392f41482e887c6b44402a641d dfcd0863f170402689d89771da0ea3ff - - -] Exception during message handling: 'module' object has no attribute 'spawn'
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6668, in live_migration
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     migrate_data=migrate_data)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5237, in live_migration
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     block_migration, migrate_data)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5330, in live_migration
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     utils.spawn(self._live_migration, context, instance, dest,
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'module' object has no attribute 'spawn'
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher

  CMD:

  nova host-evacuate-live compute8

  LOGS (/var/log/nova/nova-compute.log):

  2016-01-18 10:17:09.807 14122 ERROR oslo_messaging.rpc.dispatcher [req-53f79708-4c7c-4405-a116-ed605fc255a4 ad8ac2392f41482e887c6b44402a641d dfcd0863f170402689d89771da0ea3ff - - -] Exception during message handling: 'module' object has no attribute 'spawn'
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6668, in live_migration
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     migrate_data=migrate_data)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5237, in live_migration
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     block_migration, migrate_data)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5330, in live_migration
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     utils.spawn(self._live_migration, context, instance, dest,
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'module' object has no attribute 'spawn'
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher

  Is there something we did wrong ?

  [VM is still up and running, only state stuck in MIGRATING; had to
  nova reset-state --active vMs.]

  Thanks in advance for your help.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1535346/+subscriptions
References

[Bug 1535346] [NEW] TRACE when using nova host-migrate-live or even nova live-migration
From: David, 2016-01-18