yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1506012] Re: Gate tests randomly failing qemu-img info calls

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Doug Hellmann <doug@xxxxxxxxxxxxxxxx>
Date: Thu, 03 Dec 2015 21:32:48 -0000
Reply-to: Bug 1506012 <1506012@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1506012

Title:
  Gate tests randomly failing qemu-img info calls

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Since yesterday we see a spike in qemu-img info calls failing:

  http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiQ29tbWFuZDogZW52IExDX0FMTD1DIExBTkc9QyBxZW11LWltZyBpbmZvXCIiLCJmaWVsZHMiOltdLCJvZmZzZXQiOjAsInRpbWVmcmFtZSI6IjYwNDgwMCIsImdyYXBobW9kZSI6ImNvdW50IiwidGltZSI6eyJ1c2VyX2ludGVydmFsIjowfSwic3RhbXAiOjE0NDQ4MTQ3ODIzMTJ9

  eg

  	[req-20a22603-59e7-425d-ba06-2a9a5505b5
  6b tempest-ServersAdminTestJSON-917741884 tempest-ServersAdminTestJSON-747301663] [instance: bebbcf1a-104f-4d98-a02b-ccd20601890b] Terminating instance
  2015-10-13 17:52:40.436 ERROR nova.compute.manager [req-ef37b368-b686-42fd-bbfe-b6c3a69d063f tempest-ServersTestJSON-1340739681 tempest-ServersTestJSON-1992588300] [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d] Instance failed to spawn
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d] Traceback (most recent call last):
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2172, in _build_resources
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     yield resources
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/compute/manager.py", line 2019, in _build_and_run_instance
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     block_device_info=block_device_info)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2437, in spawn
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     admin_pass=admin_password)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 2861, in _create_image
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     instance, size, fallback_from_host)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/virt/libvirt/driver.py", line 6282, in _try_fetch_image_cache
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     size=size)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 249, in cache
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     *args, **kwargs)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 567, in create_image
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     copy_qcow2_image(base, self.path, size)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 254, in inner
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     return f(*args, **kwargs)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/virt/libvirt/imagebackend.py", line 529, in copy_qcow2_image
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     libvirt_utils.create_cow_image(base, target)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/virt/libvirt/utils.py", line 87, in create_cow_image
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     base_details = images.qemu_img_info(backing_file)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/virt/images.py", line 68, in qemu_img_info
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     preexec_fn=_qemu_resources)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/opt/stack/new/nova/nova/utils.py", line 390, in execute
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     return processutils.execute(*cmd, **kwargs)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 275, in execute
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d]     cmd=sanitized_cmd)
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d] ProcessExecutionError: Unexpected error while running command.
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d] Command: env LC_ALL=C LANG=C qemu-img info /opt/stack/data/nova/instances/_base/17293870eb69e986dd7779e8bb4f42652df5977e
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d] Exit code: -11
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d] Stdout: u''
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d] Stderr: u''
  2015-10-13 17:52:40.436 23002 ERROR nova.compute.manager [instance: 94a3b99d-b8b2-4e2a-a26a-fd7ff6e4f67d] 

  
  This spike co-incides with merging of

  commit da217205f53f9a38a573fb151898fbbeae41021d
  Author: Tristan Cacqueray <tdecacqu@xxxxxxxxxx>
  Date:   Wed Aug 5 17:17:04 2015 +0000

      virt: Use preexec_fn to ulimit qemu-img info call
      
      This uses the preexec_fn oslo.concurrency execute parameter to set user limit
      before calling qemu-img info. The process is limitted to 2 seconds of cpu
      and 1GB of virtual memory
      
      Change-Id: Ib47f1116c94c8f76d2a4a525af24439a4aa15854
      Closes-Bug: #1449062

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1506012/+subscriptions
References

[Bug 1506012] [NEW] Gate tests randomly failing qemu-img info calls
From: Daniel Berrange, 2015-10-14