← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #16919

[Bug 1337860] [NEW] VirtualInterfaceCreateException: Virtual Interface creation failed

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

After failing to launch 100 instances on size/memory allocation issue I tried launching  100 smaller instances. 
after the first failure, 2 attempts to launch 100 smaller instances come back with error on some of the instances. 
the 3ed time, I suddenly succeeded to launch all instances with no errors. 
this is reproduced 100%. 

to reproduce: 
make sure you have enough computes to run 100 tiny flavor instances. 

1. launch 100 instances with largest flavor (you should fail on memory or size). 
2. destroy all instances and run 100 tiny flavor instances - repeat this step until all instances are launched successfully. 

some of the instances will fail to be created with the below error after
the first failure, even though they should be capable of running. after
several trials we suddenly manage to run all instances (so cache issue
perhaps).


2014-07-04 14:46:19.728 15291 DEBUG nova.compute.utils [req-327fecfb-3bac-4a6d-aebe-3e06c03132e1 5a67ce69c6824e17b44bf15003ccc29f d22192179d3042a587ebd06bd6fd48d1] [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25] Virtual Interface creat
ion failed notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py:336
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25] Traceback (most recent call last):
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1191, in _run_instance
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     instance, image_meta, legacy_bdm_in_spec)
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1335, in _build_instance
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     network_info.wait(do_raise=False)
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     six.reraise(self.type_, self.value, self.tb)
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     set_access_ip=set_access_ip)
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     return function(self, context, *args, **kwargs)
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     six.reraise(self.type_, self.value, self.tb)
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     block_device_info)
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2260, in spawn
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     block_device_info)
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3670, in _create_domain_and_network
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     raise exception.VirtualInterfaceCreateException()
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25] VirtualInterfaceCreateException: Virtual Interface creation failed
2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "logs"
   https://bugs.launchpad.net/bugs/1337860/+attachment/4145535/+files/nova.tar.gz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1337860

Title:
  VirtualInterfaceCreateException: Virtual Interface creation failed

Status in OpenStack Compute (Nova):
  New

Bug description:
  After failing to launch 100 instances on size/memory allocation issue I tried launching  100 smaller instances. 
  after the first failure, 2 attempts to launch 100 smaller instances come back with error on some of the instances. 
  the 3ed time, I suddenly succeeded to launch all instances with no errors. 
  this is reproduced 100%. 

  to reproduce: 
  make sure you have enough computes to run 100 tiny flavor instances. 

  1. launch 100 instances with largest flavor (you should fail on memory or size). 
  2. destroy all instances and run 100 tiny flavor instances - repeat this step until all instances are launched successfully. 

  some of the instances will fail to be created with the below error
  after the first failure, even though they should be capable of
  running. after several trials we suddenly manage to run all instances
  (so cache issue perhaps).

  
  2014-07-04 14:46:19.728 15291 DEBUG nova.compute.utils [req-327fecfb-3bac-4a6d-aebe-3e06c03132e1 5a67ce69c6824e17b44bf15003ccc29f d22192179d3042a587ebd06bd6fd48d1] [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25] Virtual Interface creat
  ion failed notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py:336
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25] Traceback (most recent call last):
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1191, in _run_instance
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     instance, image_meta, legacy_bdm_in_spec)
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1335, in _build_instance
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     network_info.wait(do_raise=False)
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     six.reraise(self.type_, self.value, self.tb)
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     set_access_ip=set_access_ip)
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     return function(self, context, *args, **kwargs)
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     six.reraise(self.type_, self.value, self.tb)
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     block_device_info)
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2260, in spawn
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     block_device_info)
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3670, in _create_domain_and_network
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]     raise exception.VirtualInterfaceCreateException()
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25] VirtualInterfaceCreateException: Virtual Interface creation failed
  2014-07-04 14:46:19.728 15291 TRACE nova.compute.utils [instance: d2071fd8-8e09-4a43-b3c0-3ffb254b4c25]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1337860/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next