yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1346385] [NEW] libvirt: live migration stopped working after shared storage pacthes

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Vladik Romanovsky <vladik.romanovsky@xxxxxxxxxxxx>
Date: Mon, 21 Jul 2014 15:47:19 -0000
Reply-to: Bug 1346385 <1346385@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Live migration has stopped working with NFS shared storage, after this patch has been submitted:
https://review.openstack.org/#/c/91722, as one of the new checks doesn't take into account the file based shared storage.

2014-07-17 16:14:14.276 50785 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: local variable 'instance_dir' referenced before assignment
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 393, in decorated_function
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     payload)
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4465, in pre_live_migration
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     migrate_data)
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4585, in pre_live_migration
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     self._create_images_and_backing(context, instance, instance_dir,
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher UnboundLocalError: local variable 'instance_dir' referenced before assignment
2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher
2014-07-17 16:14:14.280 50785 ERROR oslo.messaging._drivers.common [-] Returning exception local variable 'instance_dir' referenced before assignment to caller
2014-07-17 16:14:14.280 50785 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line

** Affects: nova
     Importance: Undecided
     Assignee: Vladik Romanovsky (vladik-romanovsky)
         Status: New


** Tags: compute libvirt

** Changed in: nova
     Assignee: (unassigned) => Vladik Romanovsky (vladik-romanovsky)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1346385

Title:
  libvirt: live migration stopped working after shared storage pacthes

Status in OpenStack Compute (Nova):
  New

Bug description:
  Live migration has stopped working with NFS shared storage, after this patch has been submitted:
  https://review.openstack.org/#/c/91722, as one of the new checks doesn't take into account the file based shared storage.

  2014-07-17 16:14:14.276 50785 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: local variable 'instance_dir' referenced before assignment
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 393, in decorated_function
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 88, in wrapped
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     payload)
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 71, in wrapped
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 303, in decorated_function
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 290, in decorated_function
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4465, in pre_live_migration
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     migrate_data)
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 4585, in pre_live_migration
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher     self._create_images_and_backing(context, instance, instance_dir,
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher UnboundLocalError: local variable 'instance_dir' referenced before assignment
  2014-07-17 16:14:14.276 50785 TRACE oslo.messaging.rpc.dispatcher
  2014-07-17 16:14:14.280 50785 ERROR oslo.messaging._drivers.common [-] Returning exception local variable 'instance_dir' referenced before assignment to caller
  2014-07-17 16:14:14.280 50785 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1346385/+subscriptions
Follow ups

[Bug 1346385] Re: libvirt: live migration stopped working after shared storage pacthes
From: Thierry Carrez, 2014-09-05
[Bug 1346385] [NEW] libvirt: live migration stopped working after shared storage pacthes
From: Vladik Romanovsky, 2014-07-21
References

[Bug 1346385] [NEW] libvirt: live migration stopped working after shared storage pacthes
From: Vladik Romanovsky, 2014-07-21