yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1280600] [NEW] VCDriver: cold migration/resize failed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jay Lau <jay.lau.513@xxxxxxxxx>
Date: Sat, 15 Feb 2014 14:43:49 -0000
Reply-to: Bug 1280600 <1280600@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

 DC
    |
    |----Cluster1
    |          |
    |          |----9.111.249.56
    |
    |----Cluster2
               |
               |----9.111.249.49

Configure resize to different host

Case 1) 
1) Create a VM on Cluster2
2) Resize the VM
3) After resize finished, the VM goes to verify_resize.
4) confirm the resize, nova compute report error on source host.

69699f] [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932] NV-D132FDD Setting instance vm_state to ERROR
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932] Traceback (most recent call last):
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5131, in _error_out_instance_on_exception
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     yield
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2836, in _confirm_resize
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     network_info)
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 420, in confirm_migration
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     _vmops = self._get_vmops_for_compute_node(instance['node'])
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 523, in _get_vmops_for_compute_node
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     resource = self._get_resource_for_node(nodename)
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 515, in _get_resource_for_node
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     raise exception.NotFound(msg)
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932] NotFound: NV-3AB798A The resource domain-c12(Cluster1) does not exist
2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932] 
2014-02-15 17:21:23.428 4707 ERROR nova.openstack.common.rpc.amqp [req-665c6711-5353-46d7-a7b7-4d3330f41787 7566bb5312984271b2612533c04a2015 438f975992384bf5b7cf018d9569699f] Exception during message handling
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     **args)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     payload)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 299, in decorated_function
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 276, in decorated_function
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 263, in decorated_function
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2812, in confirm_resize
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     do_confirm_resize(context, instance, migration.id)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kwargs)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2810, in do_confirm_resize
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     migration=migration)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2836, in _confirm_resize
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     network_info)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 420, in confirm_migration
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     _vmops = self._get_vmops_for_compute_node(instance['node'])
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 523, in _get_vmops_for_compute_node
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     resource = self._get_resource_for_node(nodename)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 515, in _get_resource_for_node
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     raise exception.NotFound(msg)
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp NotFound: NV-3AB798A The resource domain-c12(Cluster1) does not exist
2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1280600

Title:
  VCDriver: cold migration/resize failed

Status in OpenStack Compute (Nova):
  New

Bug description:
   DC
      |
      |----Cluster1
      |          |
      |          |----9.111.249.56
      |
      |----Cluster2
                 |
                 |----9.111.249.49

  Configure resize to different host

  Case 1) 
  1) Create a VM on Cluster2
  2) Resize the VM
  3) After resize finished, the VM goes to verify_resize.
  4) confirm the resize, nova compute report error on source host.

  69699f] [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932] NV-D132FDD Setting instance vm_state to ERROR
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932] Traceback (most recent call last):
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5131, in _error_out_instance_on_exception
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     yield
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2836, in _confirm_resize
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     network_info)
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 420, in confirm_migration
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     _vmops = self._get_vmops_for_compute_node(instance['node'])
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 523, in _get_vmops_for_compute_node
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     resource = self._get_resource_for_node(nodename)
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 515, in _get_resource_for_node
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932]     raise exception.NotFound(msg)
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932] NotFound: NV-3AB798A The resource domain-c12(Cluster1) does not exist
  2014-02-15 17:21:23.191 4707 TRACE nova.compute.manager [instance: e3614b2d-ff1c-4c6e-bee6-87e7c02a4932] 
  2014-02-15 17:21:23.428 4707 ERROR nova.openstack.common.rpc.amqp [req-665c6711-5353-46d7-a7b7-4d3330f41787 7566bb5312984271b2612533c04a2015 438f975992384bf5b7cf018d9569699f] Exception during message handling
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     **args)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 90, in wrapped
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     payload)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 73, in wrapped
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 299, in decorated_function
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 276, in decorated_function
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     six.reraise(self.type_, self.value, self.tb)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 263, in decorated_function
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2812, in confirm_resize
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     do_confirm_resize(context, instance, migration.id)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     return f(*args, **kwargs)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2810, in do_confirm_resize
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     migration=migration)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2836, in _confirm_resize
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     network_info)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 420, in confirm_migration
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     _vmops = self._get_vmops_for_compute_node(instance['node'])
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 523, in _get_vmops_for_compute_node
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     resource = self._get_resource_for_node(nodename)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 515, in _get_resource_for_node
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp     raise exception.NotFound(msg)
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp NotFound: NV-3AB798A The resource domain-c12(Cluster1) does not exist
  2014-02-15 17:21:23.428 4707 TRACE nova.openstack.common.rpc.amqp

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1280600/+subscriptions
Follow ups

[Bug 1280600] [NEW] VCDriver: cold migration/resize failed
From: Jay Lau, 2014-02-15
References

[Bug 1280600] [NEW] VCDriver: cold migration/resize failed
From: Jay Lau, 2014-02-15