yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1319926] [NEW] incorrect error when failing to launch on No space left on device\n

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dafna Ron <dron@xxxxxxxxxx>
Date: Thu, 15 May 2014 16:30:31 -0000
Reply-to: Bug 1319926 <1319926@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

when we configure nova to launch instances with preallocated disk and we fail to spown an instance because we do not have enough disk space, horizon is reporting no valid host found: 
Error: Failed to launch instance "dafna": Please try again later [Error: No valid host was found. ]. 

it would be good if we can calculate the amount of space left before launching the instance from horizon (I opened a bug to nova to see if we can add a test in nova before launch - https://bugs.launchpad.net/nova/+bug/1319920). 
but, it would also be good to report that there is not enough disk space to launch the instance (as reported in compute log) which would be much clearer than no valid hosts which makes the user search for issues in the host. 

2014-05-15 19:12:57.878 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Instance failed to spa
wn
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
2014-05-15 19:12:57.878 23617 TRACE n2014-05-15 19:12:58.471 23617 INFO nova.virt.libvirt.driver [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9
a1d-c050a3abd339] Deletion of /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339 complete
2014-05-15 19:12:59.093 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Error: Unexpected erro
r while running command.
Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
Exit code: 1
Stdout: ''
Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     set_access_ip=set_access_ip)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return function(self, context, *args, **kwargs)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     six.reraise(self.type_, self.value, self.tb)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     utils.execute('fallocate', '-n', '-l', size, self.path)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return processutils.execute(*cmd, **kwargs)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     cmd=' '.join(cmd))
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] ProcessExecutionError: Unexpected error while running command.
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Exit code: 1
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stdout: ''
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] 
2014-05-15 19:13:36.350 23617 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free ram (MB): -1008
2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free disk (GB): -101
:

** Affects: horizon
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1319926

Title:
  incorrect error when failing to launch on No space left on device\n

Status in OpenStack Dashboard (Horizon):
  New

Bug description:
  when we configure nova to launch instances with preallocated disk and we fail to spown an instance because we do not have enough disk space, horizon is reporting no valid host found: 
  Error: Failed to launch instance "dafna": Please try again later [Error: No valid host was found. ]. 

  it would be good if we can calculate the amount of space left before launching the instance from horizon (I opened a bug to nova to see if we can add a test in nova before launch - https://bugs.launchpad.net/nova/+bug/1319920). 
  but, it would also be good to report that there is not enough disk space to launch the instance (as reported in compute log) which would be much clearer than no valid hosts which makes the user search for issues in the host. 

  2014-05-15 19:12:57.878 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Instance failed to spa
  wn
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
  2014-05-15 19:12:57.878 23617 TRACE n2014-05-15 19:12:58.471 23617 INFO nova.virt.libvirt.driver [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9
  a1d-c050a3abd339] Deletion of /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339 complete
  2014-05-15 19:12:59.093 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Error: Unexpected erro
  r while running command.
  Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
  Exit code: 1
  Stdout: ''
  Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     set_access_ip=set_access_ip)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return function(self, context, *args, **kwargs)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     six.reraise(self.type_, self.value, self.tb)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     utils.execute('fallocate', '-n', '-l', size, self.path)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return processutils.execute(*cmd, **kwargs)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     cmd=' '.join(cmd))
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] ProcessExecutionError: Unexpected error while running command.
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Exit code: 1
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stdout: ''
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] 
  2014-05-15 19:13:36.350 23617 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
  2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free ram (MB): -1008
  2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free disk (GB): -101
  :

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1319926/+subscriptions
Follow ups

[Bug 1319926] Re: incorrect error when failing to launch on No space left on device\n
From: Gary W. Smith, 2014-08-26
[Bug 1319926] [NEW] incorrect error when failing to launch on No space left on device\n
From: Dafna Ron, 2014-05-15
References

[Bug 1319926] [NEW] incorrect error when failing to launch on No space left on device\n
From: Dafna Ron, 2014-05-15