yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1319920] [NEW] adding space check when working with preallocation

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Dafna Ron <dron@xxxxxxxxxx>
Date: Thu, 15 May 2014 16:19:49 -0000
Reply-to: Bug 1319920 <1319920@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

when we run a large flavor on a setup configured for preallocation we fail to spown with unexpected error. 
can we add a free space test before we launch the instance (like we have for min disk size)?
to reproduce, configure nova.conf for preallocation -> launch an instance with a flavor which is bigger than the compute disk 

2014-05-15 19:12:57.183 23617 INFO nova.virt.libvirt.driver [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Creating image
2014-05-15 19:12:57.878 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Instance failed to spawn
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
2014-05-15 19:12:57.878 23617 TRACE n2014-05-15 19:12:58.471 23617 INFO nova.virt.libvirt.driver [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Deletion of /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339 complete
2014-05-15 19:12:59.093 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Error: Unexpected error while running command.
Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
Exit code: 1
Stdout: ''
Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     set_access_ip=set_access_ip)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return function(self, context, *args, **kwargs)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     six.reraise(self.type_, self.value, self.tb)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     utils.execute('fallocate', '-n', '-l', size, self.path)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return processutils.execute(*cmd, **kwargs)
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     cmd=' '.join(cmd))
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] ProcessExecutionError: Unexpected error while running command.
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Exit code: 1
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stdout: ''
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] 
(END)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1319920

Title:
  adding space check when working with preallocation

Status in OpenStack Compute (Nova):
  New

Bug description:
  when we run a large flavor on a setup configured for preallocation we fail to spown with unexpected error. 
  can we add a free space test before we launch the instance (like we have for min disk size)?
  to reproduce, configure nova.conf for preallocation -> launch an instance with a flavor which is bigger than the compute disk 

  2014-05-15 19:12:57.183 23617 INFO nova.virt.libvirt.driver [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Creating image
  2014-05-15 19:12:57.878 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Instance failed to spawn
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
  2014-05-15 19:12:57.878 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
  2014-05-15 19:12:57.878 23617 TRACE n2014-05-15 19:12:58.471 23617 INFO nova.virt.libvirt.driver [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Deletion of /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339 complete
  2014-05-15 19:12:59.093 23617 ERROR nova.compute.manager [req-1607bb0f-88a3-4888-b751-dab00e24f824 c9062d562d9f41e4a1fdce36a4f176f6 4ad766166539403189f2caca1ba306aa] [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Error: Unexpected error while running command.
  Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
  Exit code: 1
  Stdout: ''
  Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Traceback (most recent call last):
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1311, in _build_instance
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     set_access_ip=set_access_ip)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 399, in decorated_function
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return function(self, context, *args, **kwargs)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1723, in _spawn
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     LOG.exception(_('Instance failed to spawn'), instance=instance)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     six.reraise(self.type_, self.value, self.tb)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1720, in _spawn
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     block_device_info)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2246, in spawn
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     admin_pass=admin_password)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2615, in _create_image
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     project_id=instance['project_id'])
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 196, in cache
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     utils.execute('fallocate', '-n', '-l', size, self.path)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 164, in execute
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     return processutils.execute(*cmd, **kwargs)
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 193, in execute
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339]     cmd=' '.join(cmd))
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] ProcessExecutionError: Unexpected error while running command.
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Command: fallocate -n -l 171798691840 /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Exit code: 1
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stdout: ''
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] Stderr: 'fallocate: /var/lib/nova/instances/c1ad975d-6974-4539-9a1d-c050a3abd339/disk: fallocate failed: No space left on device\n'
  2014-05-15 19:12:59.093 23617 TRACE nova.compute.manager [instance: c1ad975d-6974-4539-9a1d-c050a3abd339] 
  (END)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1319920/+subscriptions
Follow ups

[Bug 1319920] Re: adding space check when working with preallocation
From: Markus Zoeller (markus_z), 2016-05-17
[Bug 1319920] [NEW] adding space check when working with preallocation
From: Dafna Ron, 2014-05-15
References

[Bug 1319920] [NEW] adding space check when working with preallocation
From: Dafna Ron, 2014-05-15