yahoo-eng-team team mailing list archive

[Fabian Wiesel <2058225@xxxxxxxxxxxxxxxxxx>]

Public bug reported:

Description
===========

This is exemplified by the tempest test in master fails with the vmwareapi driver:
tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server
Even with patch for https://review.opendev.org/c/openstack/nova/+/910627

The `ComputeManager._rebuild_default_impl` calls first destroy on the VM in both branches:
- https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L3695-L3701
And in the case of a volume backed VM with `reimage_boot_volume=True` calls `ComputeManager._rebuild_volume_backed_instance` here
- https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L3710-L3715
The function tries to detach the volume from the destroyed instance and at least in the VMware driver raises an `InstanceNotFound`, which I'd argue would be expected.
- https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L3596-L3607

Steps to reproduce
==================
* Install Devstack from master
* Run tempest test `tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server`

Or as a bash script:
```
IMAGE=$(openstack image list -c ID -f value)
ID1=$(openstack server create --flavor 1 --image $IMAGE --boot-from-volume 1 rebuild-1 -c id -f value)
ID2=$(openstack server create --flavor 1 --image $IMAGE --boot-from-volume 1 rebuild-2 -c id -f value)
# Wait for servers to be ready

# Works
openstack server rebuild --os-compute-api-version 2.93 --image $IMAGE $ID1

# Fails
openstack server rebuild --os-compute-api-version 2.93 --reimage-boot-volume --image $IMAGE $ID1

```
Expected result
===============
The test succeeds.

Actual result
=============

Environment
===========
1. Patch proposed in https://review.opendev.org/c/openstack/nova/+/909474
  + Patch proposed in https://review.opendev.org/c/openstack/nova/+/910627

2. Which hypervisor did you use? What's the version of that?

vmwareapi (VSphere 7.0.3 & ESXi 7.0.3)

2. Which storage type did you use?

vmdk on NFS 4.1

3. Which networking type did you use?

networking-nsx-t (https://github.com/sapcc/networking-nsx-t)

Logs & Configs
==============

** Affects: nova
     Importance: Undecided
         Status: New

** Summary changed:

- ComputeManager._rebuild_default_impl calls driver.detroy before driver.detach
+ ComputeManager._rebuild_default_impl calls driver.destroy before driver.detach_volume

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/2058225

Title:
  ComputeManager._rebuild_default_impl calls driver.destroy before
  driver.detach_volume

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========

  This is exemplified by the tempest test in master fails with the vmwareapi driver:
  tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server
  Even with patch for https://review.opendev.org/c/openstack/nova/+/910627

  The `ComputeManager._rebuild_default_impl` calls first destroy on the VM in both branches:
  - https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L3695-L3701
  And in the case of a volume backed VM with `reimage_boot_volume=True` calls `ComputeManager._rebuild_volume_backed_instance` here
  - https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L3710-L3715
  The function tries to detach the volume from the destroyed instance and at least in the VMware driver raises an `InstanceNotFound`, which I'd argue would be expected.
  - https://opendev.org/openstack/nova/src/branch/master/nova/compute/manager.py#L3596-L3607

  Steps to reproduce
  ==================
  * Install Devstack from master
  * Run tempest test `tempest.api.compute.servers.test_server_actions.ServerActionsV293TestJSON.test_rebuild_volume_backed_server`

  Or as a bash script:
  ```
  IMAGE=$(openstack image list -c ID -f value)
  ID1=$(openstack server create --flavor 1 --image $IMAGE --boot-from-volume 1 rebuild-1 -c id -f value)
  ID2=$(openstack server create --flavor 1 --image $IMAGE --boot-from-volume 1 rebuild-2 -c id -f value)
  # Wait for servers to be ready

  # Works
  openstack server rebuild --os-compute-api-version 2.93 --image $IMAGE $ID1

  # Fails
  openstack server rebuild --os-compute-api-version 2.93 --reimage-boot-volume --image $IMAGE $ID1

  ```
  Expected result
  ===============
  The test succeeds.

  Actual result
  =============

  Environment
  ===========
  1. Patch proposed in https://review.opendev.org/c/openstack/nova/+/909474
    + Patch proposed in https://review.opendev.org/c/openstack/nova/+/910627

  2. Which hypervisor did you use? What's the version of that?

  vmwareapi (VSphere 7.0.3 & ESXi 7.0.3)

  2. Which storage type did you use?

  vmdk on NFS 4.1

  3. Which networking type did you use?

  networking-nsx-t (https://github.com/sapcc/networking-nsx-t)

  Logs & Configs
  ==============

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/2058225/+subscriptions