yahoo-eng-team team mailing list archive

[Fabian Wiesel <fabian.wiesel@xxxxxxx>]

Public bug reported:

Description
===========
I am observing an internal server error when trying to resize the root disk.

The error is: 'NoneType' object has no attribute 'backing'
"nova/virt/vmwareapi/vmops.py", line 1314, in _resize_create_ephemerals_and_swap ds_ref = vmdk.device.backing.datastore 

As I see it, it is caused by the assumption, that the root disk is named after the UUID here:
https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vm_util.py#L634
To my knowledge, that is not generally the case, and VSphere quite happily renames it (e.g after a Storage VMotion, which may happen automatically, when the ephemeral storage is in a SDRS cluster).
In my case, the root disk is actually named <uuid>-00001.vmdk, resulting (presumably) in the following:

  root_disk= '<uuid>.vmdk'
There is no backing of the file, 
  root_device = None
And given those two variables (https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vm_util.py#L657-L658)

Resulting in   vmdk.device = None (the same as root_device) here
  https://github.com/openstack/nova/blob/stable/mitaka/nova/virt/vmwareapi/vmops.py#L1314

Steps to reproduce
==================
* Create Instance
* Trigger Storage VMotion to a different datastore (there are possibly different ways to cause the rename)
* Resize the ephemeral root device

Expected result
===============
* The VM is running with a resized root disk

Actual result
=============
The following error message:

Message:'NoneType' object has no attribute 'backing'
Code: 500
Details: File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function return function(self, context, *args, **kwargs) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4054, in finish_resize self._set_instance_obj_error_state(context, instance) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4042, in finish_resize disk_info, image_meta) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4007, in _finish_resize old_instance_type) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4002, in _finish_resize block_device_info, power_on) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 306, in finish_migration block_device_info, power_on) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 1443, in finish_migration block_device_info) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 1314, in _resize_create_ephemerals_and_swap ds_ref = vmdk.device.backing.datastore

Environment
===========

VSphere 6.0
Two ephemeral datastores on VMFS 5 (FcoE), not in a SDRS Cluster
Own Kolla Build from mitaka/stable (be033eef7296d132c8f7eb5e42203f5a3a947de6)

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1609413

Title:
  Mitaka Resize VMware Ephemeral Root Disk -> Internal Server Error 500

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  I am observing an internal server error when trying to resize the root disk.

  The error is: 'NoneType' object has no attribute 'backing'
  "nova/virt/vmwareapi/vmops.py", line 1314, in _resize_create_ephemerals_and_swap ds_ref = vmdk.device.backing.datastore 

  As I see it, it is caused by the assumption, that the root disk is named after the UUID here:
  https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vm_util.py#L634
  To my knowledge, that is not generally the case, and VSphere quite happily renames it (e.g after a Storage VMotion, which may happen automatically, when the ephemeral storage is in a SDRS cluster).
  In my case, the root disk is actually named <uuid>-00001.vmdk, resulting (presumably) in the following:

    root_disk= '<uuid>.vmdk'
  There is no backing of the file, 
    root_device = None
  And given those two variables (https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/vm_util.py#L657-L658)

  Resulting in   vmdk.device = None (the same as root_device) here
    https://github.com/openstack/nova/blob/stable/mitaka/nova/virt/vmwareapi/vmops.py#L1314

  Steps to reproduce
  ==================
  * Create Instance
  * Trigger Storage VMotion to a different datastore (there are possibly different ways to cause the rename)
  * Resize the ephemeral root device

  Expected result
  ===============
  * The VM is running with a resized root disk

  Actual result
  =============
  The following error message:

  Message:'NoneType' object has no attribute 'backing'
  Code: 500
  Details: File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 375, in decorated_function return function(self, context, *args, **kwargs) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4054, in finish_resize self._set_instance_obj_error_state(context, instance) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4042, in finish_resize disk_info, image_meta) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4007, in _finish_resize old_instance_type) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ self.force_reraise() File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise six.reraise(self.type_, self.value, self.tb) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py", line 4002, in _finish_resize block_device_info, power_on) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 306, in finish_migration block_device_info, power_on) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 1443, in finish_migration block_device_info) File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 1314, in _resize_create_ephemerals_and_swap ds_ref = vmdk.device.backing.datastore

  Environment
  ===========

  VSphere 6.0
  Two ephemeral datastores on VMFS 5 (FcoE), not in a SDRS Cluster
  Own Kolla Build from mitaka/stable (be033eef7296d132c8f7eb5e42203f5a3a947de6)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1609413/+subscriptions