yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1418155] [NEW] nova will try to create unlimited instances concurrently and timeout when resources are depleted

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Joe Talerico <jtaleric@xxxxxxxxxx>
Date: Wed, 04 Feb 2015 18:13:54 -0000
Reply-to: Bug 1418155 <1418155@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

running with --num-instances=16 I saw a couple of instances go into
ERROR State, on the hypervisor side, i saw the following issue:

2015-02-04 09:03:02.840 5077 ERROR nova.compute.manager [-] [instance: e277cf66-167f-4e81-a141-8dec12290015] Instance failed to spawn
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015] Traceback (most recent call last):
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2243, in _build_resources
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     yield resources
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2113, in _build_and_run_instance
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     block_device_info=block_device_info)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2622, in spawn
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     block_device_info, disk_info=disk_info)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4508, in _create_domain_and_network
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     power_on=power_on)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4432, in _create_domain
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     LOG.error(err)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     six.reraise(self.type_, self.value, self.tb)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4423, in _create_domain
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     domain.createWithFlags(launch_flags)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     rv = execute(f, *args, **kwargs)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     six.reraise(c, e, tb)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     rv = meth(*args, **kwargs)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 993, in createWithFlags
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015] libvirtError: error from service: CreateMachine: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]
2015-02-04 09:03:02.843 5077 AUDIT nova.compute.manager [req-663bcedd-8f56-4a84-81b1-4e7321a5f30e None] [instance: e277cf66-167f-4e81-a141-8dec12290015] Terminating instance

** Affects: nova
     Importance: Undecided
     Assignee: Dan Smith (danms)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1418155

Title:
  nova will try to create unlimited instances concurrently and timeout
  when resources are depleted

Status in OpenStack Compute (Nova):
  In Progress

Bug description:
  running with --num-instances=16 I saw a couple of instances go into
  ERROR State, on the hypervisor side, i saw the following issue:

  2015-02-04 09:03:02.840 5077 ERROR nova.compute.manager [-] [instance: e277cf66-167f-4e81-a141-8dec12290015] Instance failed to spawn
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015] Traceback (most recent call last):
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2243, in _build_resources
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     yield resources
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2113, in _build_and_run_instance
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     block_device_info=block_device_info)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2622, in spawn
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     block_device_info, disk_info=disk_info)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4508, in _create_domain_and_network
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     power_on=power_on)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4432, in _create_domain
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     LOG.error(err)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     six.reraise(self.type_, self.value, self.tb)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4423, in _create_domain
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     domain.createWithFlags(launch_flags)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     rv = execute(f, *args, **kwargs)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     six.reraise(c, e, tb)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     rv = meth(*args, **kwargs)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 993, in createWithFlags
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015] libvirtError: error from service: CreateMachine: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
  2015-02-04 09:03:02.840 5077 TRACE nova.compute.manager [instance: e277cf66-167f-4e81-a141-8dec12290015]
  2015-02-04 09:03:02.843 5077 AUDIT nova.compute.manager [req-663bcedd-8f56-4a84-81b1-4e7321a5f30e None] [instance: e277cf66-167f-4e81-a141-8dec12290015] Terminating instance

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1418155/+subscriptions
Follow ups

[Bug 1418155] Re: nova will try to create unlimited instances concurrently and timeout when resources are depleted
From: Thierry Carrez, 2015-03-20
[Bug 1418155] [NEW] nova will try to create unlimited instances concurrently and timeout when resources are depleted
From: Joe Talerico, 2015-02-04
References

[Bug 1418155] [NEW] nova will try to create unlimited instances concurrently and timeout when resources are depleted
From: Joe Talerico, 2015-02-04