yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1328455] Re: vmware: Instance going to error state while resizing

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Fri, 10 Apr 2015 09:20:50 -0000
Reply-to: Bug 1328455 <1328455@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1328455

Title:
  vmware: Instance going to error state while resizing

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  VM goes to error state in ESX while resizing due to Insufficient disk
  space on datastore.

  Ideal behaviour should bring the Instance back to Active state.

  Resize try to clone VM, if datacenter has no free space for that
  flavor size then it will give error "Insufficient disk space on
  datastore".

  Exception move VM to error state.

  Stack Trace:

  2014-06-10 09:27:57.013 26472 WARNING nova.virt.vmwareapi.vmops [req-69ff7cf8-52a6-4eb9-a179-82ac29111de7 d90ae18dfd8e43caaa2fbcc00bd5a366 683577f7c17b405686885433f960bbf6] Datacenter does not have a host / host does not have a datastore. Ignoring it.
  2014-06-10 09:27:57.647 26472 WARNING nova.virt.vmwareapi.driver [-] Task [CloneVM_Task] (returnval){
     value = "task-28954"
     _type = "Task"
   } status: error Insufficient disk space on datastore 'My ds'.
  2014-06-10 09:27:57.648 26472 ERROR nova.compute.manager [req-69ff7cf8-52a6-4eb9-a179-82ac29111de7 d90ae18dfd8e43caaa2fbcc00bd5a366 683577f7c17b405686885433f960bbf6] [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] Setting instance vm_state to ERROR
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] Traceback (most recent call last):
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 5012, in _error_out_instance_on_exception
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]     yield
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3041, in resize_instance
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]     block_device_info)
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 461, in migrate_disk_and_power_off
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]     dest, instance_type)
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1301, in migrate_disk_and_power_off
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]     self._session._wait_for_task(instance['uuid'], vm_clone_task)
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 932, in _wait_for_task
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]     ret_val = done.wait()
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]     return hubs.get_hub().switch()
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]     return self.greenlet.switch()
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e] NovaException: Insufficient disk space on datastore 'My ds'.
  2014-06-10 09:27:57.648 26472 TRACE nova.compute.manager [instance: 995cb51e-d489-4247-8661-14f38f72ef4e]
  2014-06-10 09:27:58.173 26472 WARNING nova.virt.vmwareapi.driver [-] Task [CloneVM_Task] (returnval){
     value = "task-28954"
     _type = "Task"
   } status: error Insufficient disk space on datastore 'My ds'.
  2014-06-10 09:27:58.174 26472 WARNING nova.virt.vmwareapi.driver [-] In vmwareapi:_poll_task, Got this error Trying to re-send() an already-triggered event.
  2014-06-10 09:27:58.174 26472 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.6/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall     self.f(*self.args, **self.kw)
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 989, in _poll_task
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall     done.send_exception(excep)
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 208, in send_exception
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall     return self.send(None, args)
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 150, in send
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall     assert self._result is NOT_USED, 'Trying to re-send() an already-triggered event.'
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall AssertionError: Trying to re-send() an already-triggered event.
  2014-06-10 09:27:58.174 26472 TRACE nova.openstack.common.loopingcall
  2014-06-10 09:27:58.623 26472 ERROR nova.openstack.common.rpc.amqp [req-69ff7cf8-52a6-4eb9-a179-82ac29111de7 d90ae18dfd8e43caaa2fbcc00bd5a366 683577f7c17b405686885433f960bbf6] Exception during message handling
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     **args)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     result = getattr(proxyobj, method)(ctxt, **kwargs)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 357, in decorated_function
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 93, in wrapped
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     payload)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 76, in wrapped
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     return f(self, context, *args, **kw)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 247, in decorated_function
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     pass
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 233, in decorated_function
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     function(self, context, *args, **kwargs)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 275, in decorated_function
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     e, sys.exc_info())
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 262, in decorated_function
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     return function(self, context, *args, **kwargs)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 3041, in resize_instance
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     block_device_info)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 461, in migrate_disk_and_power_off
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     dest, instance_type)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1301, in migrate_disk_and_power_off
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     self._session._wait_for_task(instance['uuid'], vm_clone_task)
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 932, in _wait_for_task
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     ret_val = done.wait()
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     return hubs.get_hub().switch()
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 177, in switch
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp     return self.greenlet.switch()
  2014-06-10 09:27:58.623 26472 TRACE nova.openstack.common.rpc.amqp NovaException: Insufficient disk space on datastore 'My ds'.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1328455/+subscriptions
References

[Bug 1328455] [NEW] vmware: Instance going to error state while resizing
From: Nileshkumar Patel, 2014-06-10