← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #17435

[Bug 1345460] [NEW] VMware: resize fails when there is more than one compute node

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Doing a nova resize on an instance when using the vsphere driver will
cause the instance to go in to error state.

The problem is that the scheduler will pick another host to spin up a
new resized instance and when the user confirms nova will fail because
its looking for the instance on the old compute.

Here is the traceback.

2014-07-16 18:14:55.271 13228 DEBUG amqp [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88
2014-07-16 18:14:55.271 13228 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70
2014-07-16 18:14:55.273 13228 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420
2014-07-16 18:14:55.299 13228 ERROR nova.openstack.common.rpc.amqp [req-3cae0e1d-2cf4-4da7-9aac-0ea5279b829d cherkasj 37af63b6867d4fe38ac312ca626ce186] Exception during message handling
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp **args)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp payload)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2683, in confirm_resize
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp do_confirm_resize(context, instance, migration_id)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2680, in do_confirm_resize
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp migration=migration)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2707, in _confirm_resize
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp network_info)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 465, in confirm_migration
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp _vmops = self._get_vmops_for_compute_node(instance['node'])
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 567, in _get_vmops_for_compute_node
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp resource = self._get_resource_for_node(nodename)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 559, in _get_resource_for_node
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp raise exception.NotFound(msg)
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp NotFound: The resource domain-c1122(compute02) does not exist
2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp
2014-07-16 18:14:57.595 13228 DEBUG nova.openstack.common.vmware.api [-] Waiting for function _invoke_api to return. func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:120
2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking _invoke_api; retry count is 0. _func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:83
2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking method <module 'nova.virt.vmwareapi.vim_util' from '/usr/lib/python2.7/dist-packages/nova/virt/vmwareap

** Affects: nova
     Importance: Critical
     Assignee: Gary Kotton (garyk)
         Status: New

** Changed in: nova
   Importance: Undecided => Critical

** Changed in: nova
     Assignee: (unassigned) => Gary Kotton (garyk)

** Changed in: nova
    Milestone: None => juno-2

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1345460

Title:
  VMware: resize fails when there is more than one compute node

Status in OpenStack Compute (Nova):
  New

Bug description:
  Doing a nova resize on an instance when using the vsphere driver will
  cause the instance to go in to error state.

  The problem is that the scheduler will pick another host to spin up a
  new resized instance and when the user confirms nova will fail because
  its looking for the instance on the old compute.

  Here is the traceback.

  2014-07-16 18:14:55.271 13228 DEBUG amqp [-] Closed channel #1 _do_close /usr/lib/python2.7/dist-packages/amqp/channel.py:88
  2014-07-16 18:14:55.271 13228 DEBUG amqp [-] using channel_id: 1 __init__ /usr/lib/python2.7/dist-packages/amqp/channel.py:70
  2014-07-16 18:14:55.273 13228 DEBUG amqp [-] Channel open _open_ok /usr/lib/python2.7/dist-packages/amqp/channel.py:420
  2014-07-16 18:14:55.299 13228 ERROR nova.openstack.common.rpc.amqp [req-3cae0e1d-2cf4-4da7-9aac-0ea5279b829d cherkasj 37af63b6867d4fe38ac312ca626ce186] Exception during message handling
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp **args)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 353, in decorated_function
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 90, in wrapped
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp payload)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 73, in wrapped
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 294, in decorated_function
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp function(self, context, *args, **kwargs)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 271, in decorated_function
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 258, in decorated_function
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2683, in confirm_resize
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp do_confirm_resize(context, instance, migration_id)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2680, in do_confirm_resize
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp migration=migration)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2707, in _confirm_resize
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp network_info)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 465, in confirm_migration
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp _vmops = self._get_vmops_for_compute_node(instance['node'])
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 567, in _get_vmops_for_compute_node
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp resource = self._get_resource_for_node(nodename)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 559, in _get_resource_for_node
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp raise exception.NotFound(msg)
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp NotFound: The resource domain-c1122(compute02) does not exist
  2014-07-16 18:14:55.299 13228 TRACE nova.openstack.common.rpc.amqp
  2014-07-16 18:14:57.595 13228 DEBUG nova.openstack.common.vmware.api [-] Waiting for function _invoke_api to return. func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:120
  2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking _invoke_api; retry count is 0. _func /usr/lib/python2.7/dist-packages/nova/openstack/common/vmware/api.py:83
  2014-07-16 18:14:57.598 13228 DEBUG nova.openstack.common.vmware.api [-] Invoking method <module 'nova.virt.vmwareapi.vim_util' from '/usr/lib/python2.7/dist-packages/nova/virt/vmwareap

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1345460/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next