yahoo-eng-team team mailing list archive

[OpenStack Infra <1764489@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.openstack.org/561704
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=23bd8f62634707fc9896a38ff4dae606c89c6c4b
Submitter: Zuul
Branch:    master

commit 23bd8f62634707fc9896a38ff4dae606c89c6c4b
Author: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date:   Mon Apr 16 18:08:56 2018 +0100

    libvirt: Report the allocated size of preallocated file based disks
    
    At present the Libvirt driver can preallocate file based disks using the
    fallocate command and importantly the `-n` option. This option allocates
    blocks on the filesystem past the initial EOF of a given file:
    
    ```
    $ touch test.img ; fallocate -n -l $(( 1024 * 1024 )) test.img
    $ ll -lah test.img
    -rw-rw-r--. 1 stack stack 0 Apr 16 13:28 test.img
    $ du -h test.img
    1.0M	test.img
    ```
    
    This results in a miscalculation of the total disk overcommit for file
    based (excluding ploop) disks as os.path.getsize is currently used to
    determine the allocated size of these disks:
    
    ```
    >>> import os
    >>> os.path.getsize('test.img')
    0
    ```
    
    Using the above example the disk overcommit would be reported as 1.0M as
    the disk appears empty yet will report a potential (virtual) size of 1.0M.
    
    However as the required blocks have already been allocated on the
    filesystem the host will report disk_available_least as missing an
    additional 1.0M, essentially doubling the allocation for each disk.
    
    To correct this the allocated size of file based (excluding ploop) disks
    is reported using `disk_size` from the `qemu-img info` command. This
    should ensure blocks allocated past the EOF of the file are taken into
    account and correctly reported as allocated.
    
    A future change should ultimately remove the use of the `-n` option with
    fallocate, however as this would not help disks that have already been
    allocated this has not been included in this change to simplify backports.
    
    Change-Id: If642e51a4e186833349a8e30b04224a3687f5594
    Closes-bug: #1764489


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1764489

Title:
  Preallocated disks are deducted twice from disk_available_least when
  using preallocated_images = space

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description
  ===========
  When using preallocated file based disks (preallocate_images = space) with the Libvirt virt driver the reported allocation for each disk appears doubled, leaving disk_available_least under reporting the amount of available resources on a compute node.

  Originally reported and debugged by mschuppert and mdbooth:
  https://bugzilla.redhat.com/show_bug.cgi?id=1554265

  Steps to reproduce
  ==================

  $ crudini --get /etc/nova/nova-cpu.conf DEFAULT preallocate_images
  space

  $ nova hypervisor-show eb515aa2-b79e-4f38-a6ea-d493a6e0657f
  +---------------------------+------------------------------------------+
  | Property                  | Value                                    |
  +---------------------------+------------------------------------------+
  [..]
  | disk_available_least      | 43                                       |
  | free_disk_gb              | 48                                       |
  [..]
  | local_gb                  | 48                                       |
  | local_gb_used             | 0                                        |
  [..]
  +---------------------------+------------------------------------------+

  $ nova flavor-show 2
  +----------------------------+----------+
  | Property                   | Value    |
  +----------------------------+----------+
  [..]
  | disk                       | 20       |
  [..]
  +----------------------------+----------+

  $ nova boot --image cirros-0.3.5-x86_64-disk --flavor 2 test
  [..]
  $ nova hypervisor-show eb515aa2-b79e-4f38-a6ea-d493a6e0657f
  +---------------------------+------------------------------------------+
  | Property                  | Value                                    |
  +---------------------------+------------------------------------------+
  [..]
  | disk_available_least      | 3                                        |
  | free_disk_gb              | 28                                       |
  [..]
  | local_gb                  | 48                                       |
  | local_gb_used             | 20                                       |
  [..]
  +---------------------------+------------------------------------------+

  Expected result
  ===============
  The allocation for each preallocated disk is reported correctly and removed from disk_available_least.

  Actual result
  =============
  The allocation for each preallocated disk appears doubled and removed from disk_available_least.

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/

     fb0b785169e5e422b06e82f2eb58e68f6d2008b3

  2. Which hypervisor did you use?
     (For example: Libvirt + KVM, Libvirt + XEN, Hyper-V, PowerKVM, ...)
     What's the version of that?

     Libvirt + KVM

  2. Which storage type did you use?
     (For example: Ceph, LVM, GPFS, ...)
     What's the version of that?

     file / qcow2

  3. Which networking type did you use?
     (For example: nova-network, Neutron with OpenVSwitch, ...)

     N/A

  Logs & Configs
  ==============
  See above.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1764489/+subscriptions