yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1535346] [NEW] TRACE when using nova host-migrate-live or even nova live-migration

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: David <dbayle@xxxxxxxxxx>
Date: Mon, 18 Jan 2016 15:20:39 -0000
Reply-to: Bug 1535346 <1535346@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Greetings,

When trying to use nova live-migration we do have this trace on the compute node logs (nova-compute.log)
The Vm stays in "MIGRATING" state, the VM is still up on the originating node, and never moves.

VERSION IN USE: ( KILO v1 )

root@compute2:~# dpkg -l | grep nova
ii  nova-common                         1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - common files
ii  nova-compute                        1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node base
ii  nova-compute-kvm                    1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt                1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node libvirt support
ii  python-nova                         1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

CMD:

nova migrate-live 6b91ffb3-bf84-48e9-b07d-99603c89055c compute4

LOGS (/var/log/nova/nova-compute.log):

2016-01-18 09:55:46.160 18571 ERROR oslo_messaging.rpc.dispatcher [req-fae37d19-0867-4796-9df3-b08be2120d15 ad8ac2392f41482e887c6b44402a641d dfcd0863f170402689d89771da0ea3ff - - -] Exception during message handling: 'module' object has no attribute 'spawn'
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6668, in live_migration
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     migrate_data=migrate_data)
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     payload)
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5237, in live_migration
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     block_migration, migrate_data)
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5330, in live_migration
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     utils.spawn(self._live_migration, context, instance, dest,
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'module' object has no attribute 'spawn'
2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher

CMD:

nova host-evacuate-live compute8

LOGS (/var/log/nova/nova-compute.log):

2016-01-18 10:17:09.807 14122 ERROR oslo_messaging.rpc.dispatcher [req-53f79708-4c7c-4405-a116-ed605fc255a4 ad8ac2392f41482e887c6b44402a641d dfcd0863f170402689d89771da0ea3ff - - -] Exception during message handling: 'module' object has no attribute 'spawn'
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6668, in live_migration
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     migrate_data=migrate_data)
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     payload)
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5237, in live_migration
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     block_migration, migrate_data)
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5330, in live_migration
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     utils.spawn(self._live_migration, context, instance, dest,
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'module' object has no attribute 'spawn'
2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher

Is there something we did wrong ?

[VM is still up and running, only state stuck in MIGRATING; had to nova
reset-state --active vMs.]

Thanks in advance for your help.

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  Greetings,
  
  When trying to use nova live-migration we do have this trace on the compute node logs (nova-compute.log)
  The Vm stays in "MIGRATING" state, the VM is still up on the originating node, and never moves.
  
  VERSION IN USE: ( KILO v1 )
  
  root@compute2:~# dpkg -l | grep nova
  ii  nova-common                         1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - common files
  ii  nova-compute                        1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                    1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node libvirt support
  ii  python-nova                         1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
  ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API
  
+ CMD:
  
- CMD: 
- 
- nova migrate-live 6b91ffb3-bf84-48e9-b07d-99603c89055c
- compute4.prod.cloud.gtcomm.net
+ nova migrate-live 6b91ffb3-bf84-48e9-b07d-99603c89055c compute4
  
  LOGS (/var/log/nova/nova-compute.log):
  
  2016-01-18 09:55:46.160 18571 ERROR oslo_messaging.rpc.dispatcher [req-fae37d19-0867-4796-9df3-b08be2120d15 ad8ac2392f41482e887c6b44402a641d dfcd0863f170402689d89771da0ea3ff - - -] Exception during message handling: 'module' object has no attribute 'spawn'
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6668, in live_migration
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     migrate_data=migrate_data)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5237, in live_migration
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     block_migration, migrate_data)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5330, in live_migration
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     utils.spawn(self._live_migration, context, instance, dest,
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'module' object has no attribute 'spawn'
- 2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher 
+ 2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher
  
- 
- CMD: 
+ CMD:
  
  nova host-evacuate-live compute8
  
  LOGS (/var/log/nova/nova-compute.log):
  
  2016-01-18 10:17:09.807 14122 ERROR oslo_messaging.rpc.dispatcher [req-53f79708-4c7c-4405-a116-ed605fc255a4 ad8ac2392f41482e887c6b44402a641d dfcd0863f170402689d89771da0ea3ff - - -] Exception during message handling: 'module' object has no attribute 'spawn'
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6668, in live_migration
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     migrate_data=migrate_data)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5237, in live_migration
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     block_migration, migrate_data)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5330, in live_migration
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     utils.spawn(self._live_migration, context, instance, dest,
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'module' object has no attribute 'spawn'
- 2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher 
+ 2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher
  
  Is there something we did wrong ?
  
  [VM is still up and running, only state stuck in MIGRATING; had to nova
  reset-state --active vMs.]
  
  Thanks in advance for your help.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535346

Title:
  TRACE when using nova host-migrate-live or even nova live-migration

Status in OpenStack Compute (nova):
  New

Bug description:
  Greetings,

  When trying to use nova live-migration we do have this trace on the compute node logs (nova-compute.log)
  The Vm stays in "MIGRATING" state, the VM is still up on the originating node, and never moves.

  VERSION IN USE: ( KILO v1 )

  root@compute2:~# dpkg -l | grep nova
  ii  nova-common                         1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - common files
  ii  nova-compute                        1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                    1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt                1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute - compute node libvirt support
  ii  python-nova                         1:2015.1.0-0ubuntu1~cloud0            all          OpenStack Compute Python libraries
  ii  python-novaclient                   1:2.22.0-0ubuntu1~cloud0              all          client library for OpenStack Compute API

  CMD:

  nova migrate-live 6b91ffb3-bf84-48e9-b07d-99603c89055c compute4

  LOGS (/var/log/nova/nova-compute.log):

  2016-01-18 09:55:46.160 18571 ERROR oslo_messaging.rpc.dispatcher [req-fae37d19-0867-4796-9df3-b08be2120d15 ad8ac2392f41482e887c6b44402a641d dfcd0863f170402689d89771da0ea3ff - - -] Exception during message handling: 'module' object has no attribute 'spawn'
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6668, in live_migration
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     migrate_data=migrate_data)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5237, in live_migration
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     block_migration, migrate_data)
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5330, in live_migration
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher     utils.spawn(self._live_migration, context, instance, dest,
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'module' object has no attribute 'spawn'
  2016-01-18 09:55:46.160 18571 TRACE oslo_messaging.rpc.dispatcher

  CMD:

  nova host-evacuate-live compute8

  LOGS (/var/log/nova/nova-compute.log):

  2016-01-18 10:17:09.807 14122 ERROR oslo_messaging.rpc.dispatcher [req-53f79708-4c7c-4405-a116-ed605fc255a4 ad8ac2392f41482e887c6b44402a641d dfcd0863f170402689d89771da0ea3ff - - -] Exception during message handling: 'module' object has no attribute 'spawn'
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6668, in live_migration
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     migrate_data=migrate_data)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 355, in decorated_function
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 343, in decorated_function
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 5237, in live_migration
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     block_migration, migrate_data)
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5330, in live_migration
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher     utils.spawn(self._live_migration, context, instance, dest,
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher AttributeError: 'module' object has no attribute 'spawn'
  2016-01-18 10:17:09.807 14122 TRACE oslo_messaging.rpc.dispatcher

  Is there something we did wrong ?

  [VM is still up and running, only state stuck in MIGRATING; had to
  nova reset-state --active vMs.]

  Thanks in advance for your help.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1535346/+subscriptions
Follow ups

[Bug 1535346] Re: Direct Snapshot in Ceph patch causes nova live-migration to break
From: Sean Dague, 2016-02-17