yahoo-eng-team team mailing list archive

[Fabian Wiesel <2055411@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========
More specifically the following tempest test in master fails:
tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server


Steps to reproduce
==================
* Install Devstack from master
* Run tempest test `tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server`

Expected result
===============
The test succeeds.

Actual result
=============
What happened instead of the expected result?
How did the issue look like?

Environment
===========
1. Git 1858cf18b940b3636e54eb5aafaf4050bdd02939 (master). So essentially this:
 https://review.opendev.org/c/openstack/nova/+/909474
As instance creation is impossible without that patch.

2. Which hypervisor did you use? What's the version of that?

vmwareapi (VSphere 7.0.3 & ESXi 7.0.3)

2. Which storage type did you use?

vmdk on NFS 4.1

3. Which networking type did you use?

networking-nsx-t (https://github.com/sapcc/networking-nsx-t)

Logs & Configs
==============

Can be found here: http://openstack-ci-
logs.global.cloud.sap/openstack/nova/1858cf18b940b3636e54eb5aafaf4050bdd02939/index.html

The critical exception for this bug report is (abbreviated and reformatted for clarity):
----
 req-7aa5ded6-ea97-4010-93c8-9e39389cbfe0 tempest-ServerActionsTestOtherA-839537081
[  865.017199] env[58735]: ERROR nova.compute.manager [instance: b4d9131c-fc91-4fd4-813b-13b4bdfe1647] 
Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 10856, in _error_out_instance_on_exception
    yield
  File "/opt/stack/nova/nova/compute/manager.py", line 6096, in _resize_instance
    disk_info = self.driver.migrate_disk_and_power_off(
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 263, in migrate_disk_and_power_off
    return self._vmops.migrate_disk_and_power_off(context, instance,
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1467, in migrate_disk_and_power_off
    self._resize_disk(instance, vm_ref, vmdk, flavor)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1398, in _resize_disk
    self._volumeops.detach_disk_from_vm(vm_ref, instance, vmdk.device)
  File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 121, in detach_disk_from_vm
    disk_key = device.key
AttributeError: 'NoneType' object has no attribute 'key'

---


The bug is actually in the function `nova.virt.vmwareapi.vm_util.get_vmdk_info` here:
https://opendev.org/openstack/nova/src/branch/master/nova/virt/vmwareapi/vm_util.py#L690

The code works with the assumption, that the root-disk is named as the instance.
This assumption breaks in several cases, but most for this test-case, the root volume is actually a cinder volume.
It will also break when the the disk gets migrated to another datastore, either through a live-migration with no shared storage, or simply automatically with SDRS..

I have an alternative implementation here: https://github.com/sapcc/nova/blob/stable/xena-m3/nova/virt/vmwareapi/vm_util.py#L997-L1034
I'll provide a bug fix from it.

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2055411

Title:
  Nova VMwareapi Resize of Volume Backed server fails

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  More specifically the following tempest test in master fails:
  tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server

  
  Steps to reproduce
  ==================
  * Install Devstack from master
  * Run tempest test `tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server`

  Expected result
  ===============
  The test succeeds.

  Actual result
  =============
  What happened instead of the expected result?
  How did the issue look like?

  Environment
  ===========
  1. Git 1858cf18b940b3636e54eb5aafaf4050bdd02939 (master). So essentially this:
   https://review.opendev.org/c/openstack/nova/+/909474
  As instance creation is impossible without that patch.

  2. Which hypervisor did you use? What's the version of that?

  vmwareapi (VSphere 7.0.3 & ESXi 7.0.3)

  2. Which storage type did you use?

  vmdk on NFS 4.1

  3. Which networking type did you use?

  networking-nsx-t (https://github.com/sapcc/networking-nsx-t)

  Logs & Configs
  ==============

  Can be found here: http://openstack-ci-
  logs.global.cloud.sap/openstack/nova/1858cf18b940b3636e54eb5aafaf4050bdd02939/index.html

  The critical exception for this bug report is (abbreviated and reformatted for clarity):
  ----
   req-7aa5ded6-ea97-4010-93c8-9e39389cbfe0 tempest-ServerActionsTestOtherA-839537081
  [  865.017199] env[58735]: ERROR nova.compute.manager [instance: b4d9131c-fc91-4fd4-813b-13b4bdfe1647] 
  Traceback (most recent call last):
    File "/opt/stack/nova/nova/compute/manager.py", line 10856, in _error_out_instance_on_exception
      yield
    File "/opt/stack/nova/nova/compute/manager.py", line 6096, in _resize_instance
      disk_info = self.driver.migrate_disk_and_power_off(
    File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 263, in migrate_disk_and_power_off
      return self._vmops.migrate_disk_and_power_off(context, instance,
    File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1467, in migrate_disk_and_power_off
      self._resize_disk(instance, vm_ref, vmdk, flavor)
    File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1398, in _resize_disk
      self._volumeops.detach_disk_from_vm(vm_ref, instance, vmdk.device)
    File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 121, in detach_disk_from_vm
      disk_key = device.key
  AttributeError: 'NoneType' object has no attribute 'key'

  ---

  
  The bug is actually in the function `nova.virt.vmwareapi.vm_util.get_vmdk_info` here:
  https://opendev.org/openstack/nova/src/branch/master/nova/virt/vmwareapi/vm_util.py#L690

  The code works with the assumption, that the root-disk is named as the instance.
  This assumption breaks in several cases, but most for this test-case, the root volume is actually a cinder volume.
  It will also break when the the disk gets migrated to another datastore, either through a live-migration with no shared storage, or simply automatically with SDRS..

  I have an alternative implementation here: https://github.com/sapcc/nova/blob/stable/xena-m3/nova/virt/vmwareapi/vm_util.py#L997-L1034
  I'll provide a bug fix from it.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2055411/+subscriptions