yahoo-eng-team team mailing list archive

[OpenStack Infra <1908408@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/nova/+/764586
Committed: https://opendev.org/openstack/nova/commit/9f260f1e6e636c991ac6252f40ff580d66b3081e
Submitter: "Zuul (22348)"
Branch:    master

commit 9f260f1e6e636c991ac6252f40ff580d66b3081e
Author: asarfaty <asarfaty@xxxxxxxxxx>
Date:   Sun Nov 29 12:27:46 2020 +0200

    Retry on vmware create_vm when it fails
    
    In some cases the logical switch is not ready when the VM is booted,
    so retry can help avoid the error.
    
    Closes-Bug: #1908408
    Change-Id: Id6e2be2b407eb08e269c35e0f89d235c67a1ab99


** Changed in: nova
       Status: Triaged => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1908408

Title:
  Need to retry vmware instance creation

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  In some cases vm boot fails with this trace:

  ERROR nova.compute.manager [None req-93f69841-12ab-42c5-b56a-83e964f4a374 tempest-TestNetworkBasicOps-721884738 tempest-TestNetworkBasicOps-721884738] [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2] Instance failed too spawn: oslo_vmware.exceptions.VimFaultException: Network interface 'VirtualE1000' uses network 'tempest-network-smoke--802933861_b25e6...71f8b (nsx.LogicalSwitch:6b0a691d-794f-42f7-b648-d67e8a10c0eb)', which is not accessible.
  Faults: ['CannotAccessNetwork']
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2] Traceback (most recent call last):
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/opt/stack/nova/nova/compute/manager.py", line 2663, in _build_resources
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     yield resources
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/opt/stack/nova/nova/compute/manager.py", line 2437, in _build_and_run_instance
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     block_device_info=block_device_info)
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 544, in spawn
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     admin_password, network_info, block_device_info)
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 753, in spawn
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     metadata)
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 304, in build_virtual_machine
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     config_spec, self._root_resource_pool)
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 1392, in create_vm
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     {'ostype': config_spec.guestId})
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     self.force_reraise()
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     six.reraise(self.type_, self.value, self.tb)
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     raise value
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 1377, in create_vm
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     task_info = session._wait_for_task(vm_create_task)
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 771, in _wait_for_task
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     return self.wait_for_task(task_ref)
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/usr/local/lib/python3.6/dist-packages/oslo_vmware/api.py", line 399, in wait_for_task
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     return evt.wait()
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/usr/local/lib/python3.6/dist-packages/eventlet/event.py", line 125, in wait
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     result = hub.switch()
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/usr/local/lib/python3.6/dist-packages/eventlet/hubs/hub.py", line 298, in switch
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     return self.greenlet.switch()
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/usr/local/lib/python3.6/dist-packages/oslo_vmware/common/loopingcall.py", line 75, in _inner
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     self.f(*self.args, **self.kw)
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]   File "/usr/local/lib/python3.6/dist-packages/oslo_vmware/api.py", line 458, in _poll_task
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2]     raise task_ex
  ERROR nova.compute.manager [instance: 26d7da79-438a-41e2-bb57-8019f90f3fa2] oslo_vmware.exceptions.VimFaultException: Network interface 'VirtualE1000' uses network 'tempest-network-smoke--802933861_b25e6...71f8b (nsx.LogicalSwitch:6b0a691d-794f-42f7-b648-d667e8a10c0eb)', which is not accessible.

  Retry will solve the problem here, and the logical switch creation
  will be completed.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1908408/+subscriptions