yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1637970] [NEW] VMware: occasional failure under load - network not ready

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Gary Kotton <1637970@xxxxxxxxxxxxxxxxxx>
Date: Mon, 31 Oct 2016 10:21:42 -0000
Reply-to: Bug 1637970 <1637970@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Executed the below testcase with 10 iterations and 7 concurrency

1. create 1 security_group
2. create 4 rules
3. create network
4. create subnet
5. create router (Distributed)
6. add router interface
6. set external gateway
5. nova boot servers
6. nova associate fip
7. verify ping from rally server
    to openstack instance


For one of the iteration, nova instance failed to spawn showing the below error.

2016-06-27 14:47:46.432 8124 ERROR nova.compute.manager [req-92a4fb34-f14f-42d0-9677-4e533d7c4c36 6bf77a5f2ea34d27948169813731737a dab0bef5c9fa40dbae556d3e6aaa13f1 - - -] [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Instance failed to spawn
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Traceback (most recent call last):
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2494, in _build_resources
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] yield resources
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2366, in _build_and_run_instance
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] block_device_info=block_device_info)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 497, in spawn
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] admin_password, network_info, block_device_info)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 784, in spawn
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] metadata)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 310, in build_virtual_machine
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] network_info)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 191, in get_vif_info
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] is_neutron, vif))
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 177, in get_vif_dict
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] ref = get_network_ref(session, cluster, vif, is_neutron)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 166, in get_network_ref
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] network_ref = get_neutron_network(session, network_name, cluster, vif)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 150, in get_neutron_network
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] session, network_name, cluster, full_match=False)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/network_util.py", line 103, in get_network_with_the_name
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] network_name, full_match)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/network_util.py", line 64, in _get_network_obj
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] "DistributedVirtualPortgroup", "config")
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 746, in _call_method
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return self.invoke_api(module, method, self.vim, *args, **kwargs)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 343, in invoke_api
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return _invoke_api(module, method, *args, **kwargs)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 122, in func
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return evt.wait()
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return hubs.get_hub().switch()
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return self.greenlet.switch()
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] idle = self.f(*self.args, **self.kw)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 95, in _func
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] result = f(*args, **kwargs)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 326, in _invoke_api
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] details=excep.details)
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] ManagedObjectNotFoundException: The object has already been deleted or has not been completely created
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Faults: [ManagedObjectNotFound]
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Details: {'obj': 'dvportgroup-4635'}
2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4]
2016-06-27 14:47:46.437 8124 INFO nova.compute.manager [req-92a4fb34-f14f-42d0-9677-4e533d7c4c36 6bf77a5f2ea34d27948169813731737a dab0bef5c9fa40dbae556d3e6aaa13f1 - - -] [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Terminating instance

** Affects: nova
     Importance: Undecided
     Assignee: Gary Kotton (garyk)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1637970

Title:
  VMware: occasional failure under load - network not ready

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Executed the below testcase with 10 iterations and 7 concurrency

  1. create 1 security_group
  2. create 4 rules
  3. create network
  4. create subnet
  5. create router (Distributed)
  6. add router interface
  6. set external gateway
  5. nova boot servers
  6. nova associate fip
  7. verify ping from rally server
      to openstack instance

  
  For one of the iteration, nova instance failed to spawn showing the below error.

  2016-06-27 14:47:46.432 8124 ERROR nova.compute.manager [req-92a4fb34-f14f-42d0-9677-4e533d7c4c36 6bf77a5f2ea34d27948169813731737a dab0bef5c9fa40dbae556d3e6aaa13f1 - - -] [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Instance failed to spawn
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Traceback (most recent call last):
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2494, in _build_resources
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] yield resources
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2366, in _build_and_run_instance
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] block_device_info=block_device_info)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 497, in spawn
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] admin_password, network_info, block_device_info)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 784, in spawn
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] metadata)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vmops.py", line 310, in build_virtual_machine
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] network_info)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 191, in get_vif_info
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] is_neutron, vif))
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 177, in get_vif_dict
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] ref = get_network_ref(session, cluster, vif, is_neutron)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 166, in get_network_ref
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] network_ref = get_neutron_network(session, network_name, cluster, vif)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vif.py", line 150, in get_neutron_network
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] session, network_name, cluster, full_match=False)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/network_util.py", line 103, in get_network_with_the_name
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] network_name, full_match)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/network_util.py", line 64, in _get_network_obj
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] "DistributedVirtualPortgroup", "config")
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 746, in _call_method
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return self.invoke_api(module, method, self.vim, *args, **kwargs)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 343, in invoke_api
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return _invoke_api(module, method, *args, **kwargs)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 122, in func
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return evt.wait()
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return hubs.get_hub().switch()
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] return self.greenlet.switch()
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 123, in _inner
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] idle = self.f(*self.args, **self.kw)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 95, in _func
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] result = f(*args, **kwargs)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 326, in _invoke_api
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] details=excep.details)
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] ManagedObjectNotFoundException: The object has already been deleted or has not been completely created
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Faults: [ManagedObjectNotFound]
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Details: {'obj': 'dvportgroup-4635'}
  2016-06-27 14:47:46.432 8124 TRACE nova.compute.manager [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4]
  2016-06-27 14:47:46.437 8124 INFO nova.compute.manager [req-92a4fb34-f14f-42d0-9677-4e533d7c4c36 6bf77a5f2ea34d27948169813731737a dab0bef5c9fa40dbae556d3e6aaa13f1 - - -] [instance: c6c1903b-d560-471a-a4bc-cc3bb52c1ed4] Terminating instance

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1637970/+subscriptions