yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1526989] [NEW] nova boot fails with "libvirtError: internal error: early end of file from monitor: possible problem: qemu: terminating on signal 15 from pid 15765"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Wed, 16 Dec 2015 23:03:10 -0000
Reply-to: Bug 1526989 <1526989@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

http://logs.openstack.org/18/258118/2/check/gate-tempest-dsvm-postgres-
full/872af9a/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-12-16_22_36_33_037

This looks to be very random:

2015-12-16 22:36:33.037 ERROR nova.compute.manager [req-706bbae7-e8b6-4c91-9d12-56d61bd2df98 tempest-ServerAddressesNegativeTestJSON-431116692 tempest-ServerAddressesNegativeTestJSON-995800820] [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] Instance failed to spawn
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] Traceback (most recent call last):
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2173, in _build_resources
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     yield resources
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2020, in _build_and_run_instance
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     block_device_info=block_device_info)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2522, in spawn
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     block_device_info=block_device_info)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4600, in _create_domain_and_network
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     xml, pause=pause, power_on=power_on)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4530, in _create_domain
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     guest.launch(pause=pause)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 141, in launch
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     self._encoded_xml, errors='ignore')
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     six.reraise(self.type_, self.value, self.tb)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 136, in launch
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     return self._domain.createWithFlags(flags)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     rv = execute(f, *args, **kwargs)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     six.reraise(c, e, tb)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     rv = meth(*args, **kwargs)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 900, in createWithFlags
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] libvirtError: internal error: early end of file from monitor: possible problem:
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] qemu: terminating on signal 15 from pid 15765
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] 
2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] 


http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22_create_domain%5C%22%20AND%20message:%5C%22libvirtError:%20internal%20error:%20early%20end%20of%20file%20from%20monitor:%20possible%20problem%5C%22%20AND%20message:%5C%22qemu:%20terminating%20on%20signal%2015%20from%20pid%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22

4 hits in the last 7 days, check and gate, all failures.

This is most likely some latent libvirt/qemu bug.

** Affects: nova
     Importance: Medium
         Status: Confirmed


** Tags: libvirt

** Tags added: libvirt

** Changed in: nova
       Status: New => Confirmed

** Changed in: nova
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1526989

Title:
  nova boot fails with "libvirtError: internal error: early end of file
  from monitor: possible problem: qemu: terminating on signal 15 from
  pid 15765"

Status in OpenStack Compute (nova):
  Confirmed

Bug description:
  http://logs.openstack.org/18/258118/2/check/gate-tempest-dsvm-
  postgres-
  full/872af9a/logs/screen-n-cpu.txt.gz?level=TRACE#_2015-12-16_22_36_33_037

  This looks to be very random:

  2015-12-16 22:36:33.037 ERROR nova.compute.manager [req-706bbae7-e8b6-4c91-9d12-56d61bd2df98 tempest-ServerAddressesNegativeTestJSON-431116692 tempest-ServerAddressesNegativeTestJSON-995800820] [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] Instance failed to spawn
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] Traceback (most recent call last):
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2173, in _build_resources
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     yield resources
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2020, in _build_and_run_instance
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     block_device_info=block_device_info)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2522, in spawn
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     block_device_info=block_device_info)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4600, in _create_domain_and_network
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     xml, pause=pause, power_on=power_on)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 4530, in _create_domain
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     guest.launch(pause=pause)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 141, in launch
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     self._encoded_xml, errors='ignore')
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 204, in __exit__
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     six.reraise(self.type_, self.value, self.tb)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/opt/stack/new/nova/nova/virt/libvirt/guest.py", line 136, in launch
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     return self._domain.createWithFlags(flags)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     result = proxy_call(self._autowrap, f, *args, **kwargs)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     rv = execute(f, *args, **kwargs)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     six.reraise(c, e, tb)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     rv = meth(*args, **kwargs)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]   File "/usr/local/lib/python2.7/dist-packages/libvirt.py", line 900, in createWithFlags
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] libvirtError: internal error: early end of file from monitor: possible problem:
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] qemu: terminating on signal 15 from pid 15765
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] 
  2015-12-16 22:36:33.037 22332 ERROR nova.compute.manager [instance: 86493959-22a5-4e5a-8226-92eb95c3c630] 

  
  http://logstash.openstack.org/#dashboard/file/logstash.json?query=message:%5C%22_create_domain%5C%22%20AND%20message:%5C%22libvirtError:%20internal%20error:%20early%20end%20of%20file%20from%20monitor:%20possible%20problem%5C%22%20AND%20message:%5C%22qemu:%20terminating%20on%20signal%2015%20from%20pid%5C%22%20AND%20tags:%5C%22screen-n-cpu.txt%5C%22

  4 hits in the last 7 days, check and gate, all failures.

  This is most likely some latent libvirt/qemu bug.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1526989/+subscriptions