yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1372708] [NEW] VMWare: vm spawn failure due to no attribute 'propSet' in concurrent case

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: zhu zhu <zhuzhubj@xxxxxxxxxx>
Date: Tue, 23 Sep 2014 01:26:24 -0000
Reply-to: Bug 1372708 <1372708@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When doing a concurrent spawn for VMs(30). Noticed some VM spawn
failures due to error below.

The reason is that during prebuild_instance, it will list all instances
of existing. But if concurrent process for spawning is happening, it is
possible that certain VM is just in process of creation in VCenter, and
the vsphere SDK only return partial object for that VM. So there is no
propSet for that VM created at the phase.

Should add check for these cases.


2014-09-14 03:53:31.311 2680 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: ObjectContent instance has no attribute 'propSet'
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     payload)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     pass
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 268, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 335, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 311, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2140, in run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     do_run_instance()
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2139, in do_run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     legacy_bdm_in_spec)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1236, in _run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     notify("error", fault=e)  # notify that build failed
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1206, in _run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     self._prebuild_instance(context, instance)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1239, in _prebuild_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     self._check_instance_exists(context, instance)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1504, in _check_instance_exists
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     if self.driver.instance_exists(instance['name']):
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/virt/driver.py", line 204, in instance_exists
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return instance_id in self.list_instances()
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 604, in list_instances
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     instances.extend(vmops.list_instances())
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 2067, in list_instances
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     lst_vm_names = self._get_valid_vms_from_retrieve_result(vms)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1851, in _get_valid_vms_from_retrieve_result
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     for prop in vm.propSet:
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher AttributeError: ObjectContent instance has no attribute 'propSet'
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: vmware

** Tags added: vmware

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1372708

Title:
  VMWare: vm spawn failure due to no attribute 'propSet' in concurrent
  case

Status in OpenStack Compute (Nova):
  New

Bug description:
  When doing a concurrent spawn for VMs(30). Noticed some VM spawn
  failures due to error below.

  The reason is that during prebuild_instance, it will list all
  instances of existing. But if concurrent process for spawning is
  happening, it is possible that certain VM is just in process of
  creation in VCenter, and the vsphere SDK only return partial object
  for that VM. So there is no propSet for that VM created at the phase.

  Should add check for these cases.

  
  2014-09-14 03:53:31.311 2680 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: ObjectContent instance has no attribute 'propSet'
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return func(*args, **kwargs)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     payload)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     pass
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 268, in decorated_function
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 335, in decorated_function
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     function(self, context, *args, **kwargs)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 311, in decorated_function
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     e, sys.exc_info())
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2140, in run_instance
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     do_run_instance()
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return f(*args, **kwargs)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2139, in do_run_instance
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     legacy_bdm_in_spec)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1236, in _run_instance
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     notify("error", fault=e)  # notify that build failed
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1206, in _run_instance
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     self._prebuild_instance(context, instance)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1239, in _prebuild_instance
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     self._check_instance_exists(context, instance)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1504, in _check_instance_exists
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     if self.driver.instance_exists(instance['name']):
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/virt/driver.py", line 204, in instance_exists
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     return instance_id in self.list_instances()
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 604, in list_instances
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     instances.extend(vmops.list_instances())
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 2067, in list_instances
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     lst_vm_names = self._get_valid_vms_from_retrieve_result(vms)
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1851, in _get_valid_vms_from_retrieve_result
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher     for prop in vm.propSet:
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher AttributeError: ObjectContent instance has no attribute 'propSet'
  2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1372708/+subscriptions
Follow ups

[Bug 1372708] Re: VMWare: vm spawn failure due to no attribute 'propSet' in concurrent case
From: Markus Zoeller (markus_z), 2016-07-05
[Bug 1372708] [NEW] VMWare: vm spawn failure due to no attribute 'propSet' in concurrent case
From: zhu zhu, 2014-09-23
References

[Bug 1372708] [NEW] VMWare: vm spawn failure due to no attribute 'propSet' in concurrent case
From: zhu zhu, 2014-09-23