yahoo-eng-team team mailing list archive

[Balazs Gibizer <balazs.gibizer@xxxxxxxx>]

Public bug reported:

Description
===========
The nova-compute service keeps a local image cache for glance images used for nova servers to avoid multiple download of the same image from glance. The disk usage of such cache is not calculated as local disk usage in nova and not reported to placement as used DISK_GB. This leads to disk over-allocation.

Also the size of that cache cannot be limited by nova configuration so
the deployer cannot reserve  disk space for that cache with
reserved_host_disk_mb config.

Steps to reproduce
==================
* Set up a single node devstack
* Create and upload an image with a not too small physical size. Like an image with 1G physical size.
* Check the current disk usage of the Host OS and configure reserved_host_disk_mb in nova-cpu.conf accordingly.
* Boot two servers from that image with a flavor, like d1 (disk=5G)
* Nova will download the glance image once to the local cache which result in a 1GB disk usage
* Nova will create two root file systems, one for each VM. Those disks initially has minimal physical disk size, but has 5G virtual size.
* At this point Nova allocated 5G + 5G of DISK_GB in placement, but due to the image in the cache the total disk usage of the two VMs + cache can be 5G + 5G + 1G, if both VMs overwrite and fills the content of its own disk.

Expected result
===============
Option A)
Nova maintains a DISK_GB allocation in placement for the images in its cache. This way the expected DISK_GB allocation in placement is 5G + 5G + 1G at the end

Option B)
Nova provides a config option to limit the maximum size of the image cache and therefore the deployer can include the maximum image cache size into the reserved_host_disk_mb during dimensioning of the disk space of the compute.

Actual result
=============
Only 5G + 5G was allocation from placement. So disk space is over-allocated by the image cache.

Environment
===========

Devstack from recent master

stack@aio:/opt/stack/nova$ git log --oneline | head -n 1
4b62c90063 Merge "Remove stale nested backport from InstancePCIRequests"

libvirt driver with file based image backend

Logs & Configs
==============
http://paste.openstack.org/show/793388/

** Affects: nova
     Importance: Undecided
         Status: New


** Tags: placement resource-tracker

** Tags added: placement resource-tracker

** Description changed:

  Description
  ===========
- The nova-compute service keeps a local image cache for glance images used for nova servers to avoid multiple download of the same image from glance. The disk usage of such cache is not calculated as local disk usage in nova and not reported to placement as used DISK_GB. This leads to disk over-allocation. 
+ The nova-compute service keeps a local image cache for glance images used for nova servers to avoid multiple download of the same image from glance. The disk usage of such cache is not calculated as local disk usage in nova and not reported to placement as used DISK_GB. This leads to disk over-allocation.
  
  Also the size of that cache cannot be limited by nova configuration so
  the deployer cannot reserve  disk space for that cache with
  reserved_host_disk_mb config.
  
  Steps to reproduce
  ==================
  * Set up a single node devstack
  * Create and upload an image with a not too small physical size. Like an image with 1G physical size.
- * Check the current disk usage of the Host OS and configure reserved_host_disk_mb in nova-cpu.conf accordingly. 
+ * Check the current disk usage of the Host OS and configure reserved_host_disk_mb in nova-cpu.conf accordingly.
  * Boot two servers from that image with a flavor, like d1 (disk=5G)
  * Nova will download the glance image once to the local cache which result in a 1GB disk usage
  * Nova will create two root file systems, one for each VM. Those disks initially has minimal physical disk size, but has 5G virtual size.
  * At this point Nova allocated 5G + 5G of DISK_GB in placement, but due to the image in the cache the total disk usage of the two VMs + cache can be 5G + 5G + 1G, if both VMs overwrite and fills the content of its own disk.
- 
  
  Expected result
  ===============
  Option A)
  Nova maintains a DISK_GB allocation in placement for the images in its cache. This way the expected DISK_GB allocation in placement is 5G + 5G + 1G at the end
  
  Option B)
  Nova provides a config option to limit the maximum size of the image cache and therefore the deployer can include the maximum image cache size into the reserved_host_disk_mb during dimensioning of the disk space of the compute.
  
- 
  Actual result
  =============
  Only 5G + 5G was allocation from placement. So disk space is over-allocated by the image cache.
- 
  
  Environment
  ===========
  
  Devstack from recent master
  
  stack@aio:/opt/stack/nova$ git log --oneline | head -n 1
  4b62c90063 Merge "Remove stale nested backport from InstancePCIRequests"
  
+ libvirt driver with file based image backend
  
  Logs & Configs
  ==============
  http://paste.openstack.org/show/793388/

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1878024

Title:
  disk usage of the nova image cache is not counted as used disk space

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  The nova-compute service keeps a local image cache for glance images used for nova servers to avoid multiple download of the same image from glance. The disk usage of such cache is not calculated as local disk usage in nova and not reported to placement as used DISK_GB. This leads to disk over-allocation.

  Also the size of that cache cannot be limited by nova configuration so
  the deployer cannot reserve  disk space for that cache with
  reserved_host_disk_mb config.

  Steps to reproduce
  ==================
  * Set up a single node devstack
  * Create and upload an image with a not too small physical size. Like an image with 1G physical size.
  * Check the current disk usage of the Host OS and configure reserved_host_disk_mb in nova-cpu.conf accordingly.
  * Boot two servers from that image with a flavor, like d1 (disk=5G)
  * Nova will download the glance image once to the local cache which result in a 1GB disk usage
  * Nova will create two root file systems, one for each VM. Those disks initially has minimal physical disk size, but has 5G virtual size.
  * At this point Nova allocated 5G + 5G of DISK_GB in placement, but due to the image in the cache the total disk usage of the two VMs + cache can be 5G + 5G + 1G, if both VMs overwrite and fills the content of its own disk.

  Expected result
  ===============
  Option A)
  Nova maintains a DISK_GB allocation in placement for the images in its cache. This way the expected DISK_GB allocation in placement is 5G + 5G + 1G at the end

  Option B)
  Nova provides a config option to limit the maximum size of the image cache and therefore the deployer can include the maximum image cache size into the reserved_host_disk_mb during dimensioning of the disk space of the compute.

  Actual result
  =============
  Only 5G + 5G was allocation from placement. So disk space is over-allocated by the image cache.

  Environment
  ===========

  Devstack from recent master

  stack@aio:/opt/stack/nova$ git log --oneline | head -n 1
  4b62c90063 Merge "Remove stale nested backport from InstancePCIRequests"

  libvirt driver with file based image backend

  Logs & Configs
  ==============
  http://paste.openstack.org/show/793388/

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1878024/+subscriptions