yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1466516] [NEW] xenproject CI intermittently fails to boot an instance with "libvirtError: internal error: libxenlight failed to create new domain"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Thu, 18 Jun 2015 13:46:42 -0000
Reply-to: Bug 1466516 <1466516@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.xenproject.org/06/192406/2/check/dsvm-tempest-
xen/216f7be/logs/screen-n-cpu.txt.gz

2015-06-18 01:30:32.511 ERROR nova.compute.manager [req-999af61f-7db0-4164-8528-7bd7917084d7 ServersTestJSON-1414905229 ServersTestJSON-1459353747] [instance: db65700a-2792-4395-a7c8-1941887b5ddc] Instance failed to spawn
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc] Traceback (most recent call last):
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2113, in _build_resources
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     yield resources
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1985, in _build_and_run_instance
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     block_device_info=block_device_info)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2435, in spawn
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     block_device_info=block_device_info)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4462, in _create_domain_and_network
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     xml, pause=pause, power_on=power_on)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4392, in _create_domain
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     guest.launch(pause=pause)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 109, in launch
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     self._encoded_xml, errors='ignore')
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     six.reraise(self.type_, self.value, self.tb)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 104, in launch
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     return self._domain.createWithFlags(flags)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     rv = execute(f, *args, **kwargs)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     six.reraise(c, e, tb)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     rv = meth(*args, **kwargs)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1037, in createWithFlags
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc] libvirtError: internal error: libxenlight failed to create new domain 'instance-0000004e'
2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc] 

The change being tested has nothing to do with this code.

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: xenserver

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1466516

Title:
  xenproject CI intermittently fails to boot an instance with
  "libvirtError: internal error: libxenlight failed to create new
  domain"

Status in OpenStack Compute (Nova):
  New

Bug description:
  http://logs.openstack.xenproject.org/06/192406/2/check/dsvm-tempest-
  xen/216f7be/logs/screen-n-cpu.txt.gz

  2015-06-18 01:30:32.511 ERROR nova.compute.manager [req-999af61f-7db0-4164-8528-7bd7917084d7 ServersTestJSON-1414905229 ServersTestJSON-1459353747] [instance: db65700a-2792-4395-a7c8-1941887b5ddc] Instance failed to spawn
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc] Traceback (most recent call last):
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2113, in _build_resources
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     yield resources
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/compute/manager.py", line 1985, in _build_and_run_instance
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     block_device_info=block_device_info)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2435, in spawn
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     block_device_info=block_device_info)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4462, in _create_domain_and_network
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     xml, pause=pause, power_on=power_on)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4392, in _create_domain
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     guest.launch(pause=pause)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 109, in launch
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     self._encoded_xml, errors='ignore')
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     six.reraise(self.type_, self.value, self.tb)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 104, in launch
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     return self._domain.createWithFlags(flags)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     rv = execute(f, *args, **kwargs)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     six.reraise(c, e, tb)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     rv = meth(*args, **kwargs)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]   File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1037, in createWithFlags
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc] libvirtError: internal error: libxenlight failed to create new domain 'instance-0000004e'
  2015-06-18 01:30:32.511 15191 ERROR nova.compute.manager [instance: db65700a-2792-4395-a7c8-1941887b5ddc] 

  The change being tested has nothing to do with this code.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1466516/+subscriptions
Follow ups

[Bug 1466516] [NEW] xenproject CI intermittently fails to boot an instance with "libvirtError: internal error: libxenlight failed to create new domain"
From: Matt Riedemann, 2015-06-18
References

[Bug 1466516] [NEW] xenproject CI intermittently fails to boot an instance with "libvirtError: internal error: libxenlight failed to create new domain"
From: Matt Riedemann, 2015-06-18