← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #54704

[Bug 1610015] [NEW] Creating an instance using images_type=lvm fails on ephemeral volume creation

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:


Steps to reproduce:

# grep -B1 images /etc/nova/nova.conf
[libvirt]
images_type = lvm
images_volume_group = vgroot

# nova boot --flavor <flavor id> --image <image_id> --key-name
id_rsa_test --security-groups default --nic net-id=<net_id> test

Expected result: instance will boot
Actual result: instance spawn fail. 

Resulting /var/log/nova/nova-compute.log (on hypervisor)

Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [req-d3efb66e-b216-4944-9b61-a0afeb82048e 93c084ec050a4ffc828883d7dbba9dc6 0b01b979930c47369f5933ef06abb768 - - -] [instance: 596d8647-4093-4e29-aac9-51d986d34e68] Instance failed to spawn
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68] Traceback (most recent call last):
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2218, in _build_resources
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     yield resources
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     block_device_info=block_device_info)
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2761, in spawn
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     admin_pass=admin_password)
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3201, in _create_image
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     ephemeral_size=ephemeral_gb)
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 254, in cache
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     if size > self.get_disk_size(base):
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 308, in get_disk_size
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     return disk.get_disk_size(name)
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 147, in get_disk_size
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     return images.qemu_img_info(path).virtual_size
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 50, in qemu_img_info
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     raise exception.DiskNotFound(location=path)
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68] DiskNotFound: No disk at /var/lib/nova/instances/_base/ephemeral_5_40d1d2c
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.295 10477 INFO nova.compute.manager [req-d3efb66e-b216-4944-9b61-a0afeb82048e 93c084ec050a4ffc828883d7dbba9dc6 0b01b979930c47369f5933ef06abb768 - - -] [instance: 596d8647-4093-4e29-aac9-51d986d34e68] Terminating instance
Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.297 10477 DEBUG nova.compute.manager [req-d3efb66e-b216-4944-9b61-a0afeb82048e 93c084ec050a4ffc828883d7dbba9dc6 0b01b979930c47369f5933ef06abb768 - - -] [instance: 596d8647-4093-4e29-aac9-51d986d34e68] Start destroying the instance on the hypervisor. _shutdown_instance /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2330


Workaround: # touch /var/lib/nova/instances/_base/ephemeral_5_40d1d2c
on the hypervisor will address the issue and allow the instance to start. 


Additional info:

# nova flavor-show c2-r4-d5
+----------------------------+--------------------------------------+
| Property                   | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 5                                    |
| disk                       | 10                                   |
| extra_specs                | {}                                   |
| id                         | 106ab90e-dfd0-4bbf-a232-cfaee5a34cc6 |
| name                       | c2-r4-d5                             |
| os-flavor-access:is_public | True                                 |
| ram                        | 4096                                 |
| rxtx_factor                | 1.0                                  |
| swap                       | 2048                                 |
| vcpus                      | 2                                    |
+----------------------------+--------------------------------------+


Hypervisor host:
# uname -a
Linux enc01-c03 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
# lsb_release -sc
xenial

# dpkg -l | grep nova
ii  nova-common                        2:13.0.0-0ubuntu5                   all          OpenStack Compute - common files
ii  nova-compute                       2:13.0.0-0ubuntu5                   all          OpenStack Compute - compute node base
ii  nova-compute-kvm                   2:13.0.0-0ubuntu5                   all          OpenStack Compute - compute node (KVM)
ii  nova-compute-libvirt               2:13.0.0-0ubuntu5                   all          OpenStack Compute - compute node libvirt support
ii  python-nova                        2:13.0.0-0ubuntu5                   all          OpenStack Compute Python libraries
ii  python-novaclient                  2:3.3.1-2                           all          client library for OpenStack Compute API - Python 2.7


# vgdisplay
  --- Volume group ---
  VG Name               vgroot
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  82
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                2
  Open LV               2
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               68.32 GiB
  PE Size               4.00 MiB
  Total PE              17491
  Alloc PE / Size       10841 / 42.35 GiB
  Free  PE / Size       6650 / 25.98 GiB
  VG UUID               eNzdGr-0g5b-YDyE-myxL-AdCR-g94Q-G5eUSR

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1610015

Title:
  Creating an instance using images_type=lvm fails on ephemeral volume
  creation

Status in OpenStack Compute (nova):
  New

Bug description:
  
  Steps to reproduce:

  # grep -B1 images /etc/nova/nova.conf
  [libvirt]
  images_type = lvm
  images_volume_group = vgroot

  # nova boot --flavor <flavor id> --image <image_id> --key-name
  id_rsa_test --security-groups default --nic net-id=<net_id> test

  Expected result: instance will boot
  Actual result: instance spawn fail. 

  Resulting /var/log/nova/nova-compute.log (on hypervisor)

  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [req-d3efb66e-b216-4944-9b61-a0afeb82048e 93c084ec050a4ffc828883d7dbba9dc6 0b01b979930c47369f5933ef06abb768 - - -] [instance: 596d8647-4093-4e29-aac9-51d986d34e68] Instance failed to spawn
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68] Traceback (most recent call last):
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2218, in _build_resources
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     yield resources
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     block_device_info=block_device_info)
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2761, in spawn
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     admin_pass=admin_password)
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3201, in _create_image
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     ephemeral_size=ephemeral_gb)
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 254, in cache
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     if size > self.get_disk_size(base):
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 308, in get_disk_size
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     return disk.get_disk_size(name)
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 147, in get_disk_size
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     return images.qemu_img_info(path).virtual_size
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]   File "/usr/lib/python2.7/dist-packages/nova/virt/images.py", line 50, in qemu_img_info
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]     raise exception.DiskNotFound(location=path)
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68] DiskNotFound: No disk at /var/lib/nova/instances/_base/ephemeral_5_40d1d2c
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.294 10477 ERROR nova.compute.manager [instance: 596d8647-4093-4e29-aac9-51d986d34e68]
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.295 10477 INFO nova.compute.manager [req-d3efb66e-b216-4944-9b61-a0afeb82048e 93c084ec050a4ffc828883d7dbba9dc6 0b01b979930c47369f5933ef06abb768 - - -] [instance: 596d8647-4093-4e29-aac9-51d986d34e68] Terminating instance
  Aug  4 22:06:32 ubuntu nova-compute[10477]: 2016-08-04 22:06:32.297 10477 DEBUG nova.compute.manager [req-d3efb66e-b216-4944-9b61-a0afeb82048e 93c084ec050a4ffc828883d7dbba9dc6 0b01b979930c47369f5933ef06abb768 - - -] [instance: 596d8647-4093-4e29-aac9-51d986d34e68] Start destroying the instance on the hypervisor. _shutdown_instance /usr/lib/python2.7/dist-packages/nova/compute/manager.py:2330

  
  Workaround: # touch /var/lib/nova/instances/_base/ephemeral_5_40d1d2c
  on the hypervisor will address the issue and allow the instance to start. 

  
  Additional info:

  # nova flavor-show c2-r4-d5
  +----------------------------+--------------------------------------+
  | Property                   | Value                                |
  +----------------------------+--------------------------------------+
  | OS-FLV-DISABLED:disabled   | False                                |
  | OS-FLV-EXT-DATA:ephemeral  | 5                                    |
  | disk                       | 10                                   |
  | extra_specs                | {}                                   |
  | id                         | 106ab90e-dfd0-4bbf-a232-cfaee5a34cc6 |
  | name                       | c2-r4-d5                             |
  | os-flavor-access:is_public | True                                 |
  | ram                        | 4096                                 |
  | rxtx_factor                | 1.0                                  |
  | swap                       | 2048                                 |
  | vcpus                      | 2                                    |
  +----------------------------+--------------------------------------+

  
  Hypervisor host:
  # uname -a
  Linux enc01-c03 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
  # lsb_release -sc
  xenial

  # dpkg -l | grep nova
  ii  nova-common                        2:13.0.0-0ubuntu5                   all          OpenStack Compute - common files
  ii  nova-compute                       2:13.0.0-0ubuntu5                   all          OpenStack Compute - compute node base
  ii  nova-compute-kvm                   2:13.0.0-0ubuntu5                   all          OpenStack Compute - compute node (KVM)
  ii  nova-compute-libvirt               2:13.0.0-0ubuntu5                   all          OpenStack Compute - compute node libvirt support
  ii  python-nova                        2:13.0.0-0ubuntu5                   all          OpenStack Compute Python libraries
  ii  python-novaclient                  2:3.3.1-2                           all          client library for OpenStack Compute API - Python 2.7

  
  # vgdisplay
    --- Volume group ---
    VG Name               vgroot
    System ID
    Format                lvm2
    Metadata Areas        1
    Metadata Sequence No  82
    VG Access             read/write
    VG Status             resizable
    MAX LV                0
    Cur LV                2
    Open LV               2
    Max PV                0
    Cur PV                1
    Act PV                1
    VG Size               68.32 GiB
    PE Size               4.00 MiB
    Total PE              17491
    Alloc PE / Size       10841 / 42.35 GiB
    Free  PE / Size       6650 / 25.98 GiB
    VG UUID               eNzdGr-0g5b-YDyE-myxL-AdCR-g94Q-G5eUSR

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1610015/+subscriptions
  Thread Previous • Date Previous • • Thread Next