yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1630912] [NEW] stable/newton: unable to spin up a kvm instance

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Gary Kotton <1630912@xxxxxxxxxxxxxxxxxx>
Date: Thu, 06 Oct 2016 09:26:54 -0000
Reply-to: Bug 1630912 <1630912@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Happens randowmly:

2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [req-50d2dc6d-072e-423e-9de0-096ba307411f admin demo] [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4] Instance failed to spawn
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4] Traceback (most recent call last):
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/compute/manager.py", line 2078, in _build_resources
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     yield resources
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/compute/manager.py", line 1920, in _build_and_run_instance
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     block_device_info=block_device_info)
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2571, in spawn
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     admin_pass=admin_password)
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2960, in _create_image
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     image_id=disk_images['kernel_id'])
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 218, in cache
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     *args, **kwargs)
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 504, in create_image
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     prepare_template(target=base, *args, **kwargs)
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 264, in inner
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     do_log=False, semaphores=semaphores, delay=delay):
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     return self.gen.next()
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 216, in lock
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     ext_lock.acquire(delay=delay)
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/local/lib/python2.7/dist-packages/fasteners/process_lock.py", line 151, in acquire
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     self._do_open()
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/local/lib/python2.7/dist-packages/fasteners/process_lock.py", line 123, in _do_open
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     self.lockfile = open(self.path, 'a')
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4] IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/locks/nova-273d5645757056cdd056de4cfe9f121b9eee6ae3'
2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]

** Affects: nova
     Importance: Critical
         Status: New

** Changed in: nova
   Importance: Undecided => Critical

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1630912

Title:
  stable/newton: unable to spin up a kvm instance

Status in OpenStack Compute (nova):
  New

Bug description:
  Happens randowmly:

  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [req-50d2dc6d-072e-423e-9de0-096ba307411f admin demo] [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4] Instance failed to spawn
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4] Traceback (most recent call last):
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/compute/manager.py", line 2078, in _build_resources
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     yield resources
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/compute/manager.py", line 1920, in _build_and_run_instance
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     block_device_info=block_device_info)
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2571, in spawn
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     admin_pass=admin_password)
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2960, in _create_image
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     image_id=disk_images['kernel_id'])
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 218, in cache
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     *args, **kwargs)
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/opt/stack/nova/nova/virt/libvirt/imagebackend.py", line 504, in create_image
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     prepare_template(target=base, *args, **kwargs)
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 264, in inner
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     do_log=False, semaphores=semaphores, delay=delay):
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     return self.gen.next()
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 216, in lock
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     ext_lock.acquire(delay=delay)
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/local/lib/python2.7/dist-packages/fasteners/process_lock.py", line 151, in acquire
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     self._do_open()
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]   File "/usr/local/lib/python2.7/dist-packages/fasteners/process_lock.py", line 123, in _do_open
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]     self.lockfile = open(self.path, 'a')
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4] IOError: [Errno 13] Permission denied: '/opt/stack/data/nova/instances/locks/nova-273d5645757056cdd056de4cfe9f121b9eee6ae3'
  2016-10-06 20:22:18.100 1931 ERROR nova.compute.manager [instance: 04709c94-0ba3-4312-ad1c-5739e5600bf4]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1630912/+subscriptions
Follow ups

[Bug 1630912] Re: stable/newton: unable to spin up a kvm instance
From: Sujai, 2016-11-07