yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1319926] Re: incorrect error when failing to launch on No space left on device\n

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "Gary W. Smith" <gary.w.smith@xxxxxx>
Date: Mon, 25 Aug 2014 23:55:23 -0000
Reply-to: Bug 1319926 <1319926@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This is the generic error that nova returns when it is unable to launch
the instance. Changing nova to providing a more precise  message is
covered in the bug you created
(https://bugs.launchpad.net/nova/+bug/1319920).

It is my understanding that it is not feasible in general for the UI to
know in advance which node that the nova scheduler will eventually
create the instance on and to calculate the remaining space on that
node, and thus to prevent the user from attempting it in the first
place. If that understanding is incorrect, then feel free to update this
bug.

** Changed in: horizon
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Dashboard (Horizon).
https://bugs.launchpad.net/bugs/1319926

Title:
  incorrect error when failing to launch on No space left on device\n

Status in OpenStack Dashboard (Horizon):
  Invalid

Bug description:
  when we configure nova to launch instances with preallocated disk and we fail to spown an instance because we do not have enough disk space, horizon is reporting no valid host found: 
  Error: Failed to launch instance "dafna": Please try again later [Error: No valid host was found. ]. 

  it would be good if we can calculate the amount of space left before launching the instance from horizon (I opened a bug to nova to see if we can add a test in nova before launch - https://bugs.launchpad.net/nova/+bug/1319920). 
  but, it would also be good to report that there is not enough disk space to launch the instance (as reported in compute log) which would be much clearer than no valid hosts which makes the user search for issues in the host. 

  2014-05-15 19:12:57.878 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Instance failed to spa
  wn
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
  2014-05-15 19:12:57.878 23617 TRACE n2014-05-15 19:12:58.471 23617 INFO nova.virt.libvirt.driver [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9
  a1d-c050a3abd339] Deletion of /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339 complete
  2014-05-15 19:12:59.093 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Error: Unexpected erro
  r while running command.
  Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
  Exit code: 1
  Stdout: ''
  Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     set_access_ip=set_access_ip)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return function(self, context, *args, **kwargs)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     six.reraise(self.type_, self.value, self.tb)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     utils.execute('fallocate', '-n', '-l', size, self.path)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return processutils.execute(*cmd, **kwargs)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     cmd=' '.join(cmd))
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] ProcessExecutionError: Unexpected error while running command.
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Exit code: 1
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stdout: ''
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] 
  2014-05-15 19:13:36.350 23617 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
  2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free ram (MB): -1008
  2014-05-15 19:13:36.493 23617 AUDIT nova.compute.resource_tracker [-] Free disk (GB): -101
  :

To manage notifications about this bug go to:
https://bugs.launchpad.net/horizon/+bug/1319926/+subscriptions
References

[Bug 1319926] [NEW] incorrect error when failing to launch on No space left on device\n
From: Dafna Ron, 2014-05-15