yahoo-eng-team team mailing list archive

[Ryan Hsu <rhsu@xxxxxxxxxx>]

Public bug reported:

The VMware Minesweeper CI is occasionally seeing a "concurrent
modification" failure when creating an instance. It's possible that this
is the cause of a race condition in the VC Driver.

Task [CreateVM_Task] (returnval){
   value = "task-932"
   _type = "Task"
 } status: success _poll_task /opt/stack/nova/nova/virt/vmwareapi/driver.py:968

Created VM on the ESX host _execute_create_vm
/opt/stack/nova/nova/virt/vmwareapi/vmops.py:305

Task [ReconfigVM_Task] (returnval){
   value = "task-939"
   _type = "Task"
 } status: error Cannot complete operation due to concurrent modification by another operation.

Fault Text not matched.

Setting instance vm_state to ERROR

Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 5581, in _error_out_instance_on_exception
    yield
  File "/opt/stack/nova/nova/compute/manager.py", line 2562, in rebuild_instance
    self._rebuild_default_impl(**kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 2417, in _rebuild_default_impl
    block_device_info=new_block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 619, in spawn
    admin_password, network_info, block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 593, in spawn
    root_gb_in_kb, linked_clone)
  File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 75, in attach_disk_to_vm
    self._session._wait_for_task(reconfig_task)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 948, in _wait_for_task
    ret_val = done.wait()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
VMwareDriverException: Cannot complete operation due to concurrent modification by another operation.

Full log for an affected run here: http://10.148.255.241/logs/83375/2/

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: vmware

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1310810

Title:
  VMware: Cannot complete operation due to concurrent modification by
  another operation

Status in OpenStack Compute (Nova):
  New

Bug description:
  The VMware Minesweeper CI is occasionally seeing a "concurrent
  modification" failure when creating an instance. It's possible that
  this is the cause of a race condition in the VC Driver.

  Task [CreateVM_Task] (returnval){
     value = "task-932"
     _type = "Task"
   } status: success _poll_task /opt/stack/nova/nova/virt/vmwareapi/driver.py:968

  Created VM on the ESX host _execute_create_vm
  /opt/stack/nova/nova/virt/vmwareapi/vmops.py:305

  Task [ReconfigVM_Task] (returnval){
     value = "task-939"
     _type = "Task"
   } status: error Cannot complete operation due to concurrent modification by another operation.

  Fault Text not matched.

  Setting instance vm_state to ERROR

  Traceback (most recent call last):
    File "/opt/stack/nova/nova/compute/manager.py", line 5581, in _error_out_instance_on_exception
      yield
    File "/opt/stack/nova/nova/compute/manager.py", line 2562, in rebuild_instance
      self._rebuild_default_impl(**kwargs)
    File "/opt/stack/nova/nova/compute/manager.py", line 2417, in _rebuild_default_impl
      block_device_info=new_block_device_info)
    File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 619, in spawn
      admin_password, network_info, block_device_info)
    File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 593, in spawn
      root_gb_in_kb, linked_clone)
    File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 75, in attach_disk_to_vm
      self._session._wait_for_task(reconfig_task)
    File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 948, in _wait_for_task
      ret_val = done.wait()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
      return hubs.get_hub().switch()
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
      return self.greenlet.switch()
  VMwareDriverException: Cannot complete operation due to concurrent modification by another operation.

  Full log for an affected run here: http://10.148.255.241/logs/83375/2/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1310810/+subscriptions