yahoo-eng-team team mailing list archive

[Ryan Hsu <rhsu@xxxxxxxxxx>]

Public bug reported:

Some instances are failing to spawn with the following error message:

VMwareDriverException: Cannot complete operation due to concurrent
modification by another operation.

It's possible this is due to a race condition in the VC driver as this
does not happen frequently. This was encountered a few times by the
Minesweeper CI. Affected builds include:

http://208.91.1.172/logs/81905/6
http://208.91.1.172/logs/80220/3

The Traceback seen in the scheduler log is:

Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 1306, in _build_instance
    set_access_ip=set_access_ip)
  File "/opt/stack/nova/nova/compute/manager.py", line 394, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 1718, in _spawn
    LOG.exception(_('Instance failed to spawn'), instance=instance)
  File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/compute/manager.py", line 1715, in _spawn
    block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 611, in spawn
    admin_password, network_info, block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 593, in spawn
    root_gb_in_kb, linked_clone)
  File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 75, in attach_disk_to_vm
    self._session._wait_for_task(reconfig_task)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 940, in _wait_for_task
    ret_val = done.wait()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
VMwareDriverException: Cannot complete operation due to concurrent modification by another operation.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: driver vmware

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1296948

Title:
  VMware: Instance fails to spawn due to "Concurrent modification by
  another operation"

Status in OpenStack Compute (Nova):
  New

Bug description:
  Some instances are failing to spawn with the following error message:

  VMwareDriverException: Cannot complete operation due to concurrent
  modification by another operation.

  It's possible this is due to a race condition in the VC driver as this
  does not happen frequently. This was encountered a few times by the
  Minesweeper CI. Affected builds include:

  http://208.91.1.172/logs/81905/6
  http://208.91.1.172/logs/80220/3

  The Traceback seen in the scheduler log is:

  Traceback (most recent call last):
    File "/opt/stack/nova/nova/compute/manager.py", line 1306, in _build_instance
      set_access_ip=set_access_ip)
    File "/opt/stack/nova/nova/compute/manager.py", line 394, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/opt/stack/nova/nova/compute/manager.py", line 1718, in _spawn
      LOG.exception(_('Instance failed to spawn'), instance=instance)
    File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/nova/nova/compute/manager.py", line 1715, in _spawn
      block_device_info)
    File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 611, in spawn
      admin_password, network_info, block_device_info)
    File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 593, in spawn
      root_gb_in_kb, linked_clone)
    File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 75, in attach_disk_to_vm
      self._session._wait_for_task(reconfig_task)
    File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 940, in _wait_for_task
      ret_val = done.wait()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
      return hubs.get_hub().switch()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
      return self.greenlet.switch()
  VMwareDriverException: Cannot complete operation due to concurrent modification by another operation.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1296948/+subscriptions