yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1815792] [NEW] Unable to delete instance when volume detached at the same time

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Gary Kotton <1815792@xxxxxxxxxxxxxxxxxx>
Date: Wed, 13 Feb 2019 18:12:47 -0000
Reply-to: Bug 1815792 <1815792@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When using K8s above OpenStack there is a race condition when a
persistant volume will be deleted at the same time that a instance using
that volume is deleted. This results in the instance going into an error
state.

201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] Traceback (most recent call last):
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2478, in do_terminate_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._delete_instance(context, instance, bdms, quotas)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     rv = f(*args, **kwargs)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2441, in _delete_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     quotas.rollback()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.force_reraise()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     six.reraise(self.type_, self.value, self.tb)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2405, in _delete_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._shutdown_instance(context, instance, bdms)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._detach_volume_vmdk(connection_info, instance)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 609, in _detach_volume_vmdk
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.detach_disk_from_vm(vm_ref, instance, device)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 157, in detach_disk_from_vm
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     vm_util.reconfigure_vm(self._session, vm_ref, vmdk_detach_config_spec)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vm_util.py", line 2681, in reconfigure_vm
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     session._wait_for_task(reconfig_task)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 719, in _wait_for_task
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     return self.wait_for_task(task_ref)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 398, in wait_for_task
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     return evt.wait()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     return hubs.get_hub().switch()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     return self.greenlet.switch()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.f(*self.args, **self.kw)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 454, in _poll_task
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     raise task_ex
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] VimFaultException: Invalid configuration for device '0'.
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] Faults: ['InvalidDeviceSpec']
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]

201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2305, in _shutdown_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     requested_networks)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.force_reraise()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     six.reraise(self.type_, self.value, self.tb)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2292, in _shutdown_instance
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     block_device_info)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 564, in destroy
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._detach_instance_volumes(instance, block_device_info)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 539, in _detach_instance_volumes
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     instance=instance)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.force_reraise()
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     six.reraise(self.type_, self.value, self.tb)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 528, in _detach_instance_volumes
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     disk.get('device_name'))
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 470, in detach_volume
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._volumeops.detach_volume(connection_info, instance)
201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 663, in detach_volume

** Affects: nova
     Importance: Undecided
     Assignee: Gary Kotton (garyk)
         Status: In Progress

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1815792

Title:
  Unable to delete instance when volume detached at the same time

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  When using K8s above OpenStack there is a race condition when a
  persistant volume will be deleted at the same time that a instance
  using that volume is deleted. This results in the instance going into
  an error state.

  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] Traceback (most recent call last):
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2478, in do_terminate_instance
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._delete_instance(context, instance, bdms, quotas)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 154, in inner
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     rv = f(*args, **kwargs)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2441, in _delete_instance
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     quotas.rollback()
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.force_reraise()
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     six.reraise(self.type_, self.value, self.tb)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2405, in _delete_instance
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._shutdown_instance(context, instance, bdms)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._detach_volume_vmdk(connection_info, instance)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 609, in _detach_volume_vmdk
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.detach_disk_from_vm(vm_ref, instance, device)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 157, in detach_disk_from_vm
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     vm_util.reconfigure_vm(self._session, vm_ref, vmdk_detach_config_spec)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vm_util.py", line 2681, in reconfigure_vm
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     session._wait_for_task(reconfig_task)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 719, in _wait_for_task
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     return self.wait_for_task(task_ref)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 398, in wait_for_task
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     return evt.wait()
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     return hubs.get_hub().switch()
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     return self.greenlet.switch()
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.f(*self.args, **self.kw)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 454, in _poll_task
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     raise task_ex
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] VimFaultException: Invalid configuration for device '0'.
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c] Faults: ['InvalidDeviceSpec']
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]

  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2305, in _shutdown_instance
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     requested_networks)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.force_reraise()
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     six.reraise(self.type_, self.value, self.tb)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2292, in _shutdown_instance
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     block_device_info)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 564, in destroy
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._detach_instance_volumes(instance, block_device_info)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 539, in _detach_instance_volumes
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     instance=instance)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self.force_reraise()
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     six.reraise(self.type_, self.value, self.tb)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 528, in _detach_instance_volumes
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     disk.get('device_name'))
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 470, in detach_volume
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]     self._volumeops.detach_volume(connection_info, instance)
  201902131136/compute01/logs/nova/nova-compute.log:2019-02-13 08:29:45.432 26114 ERROR nova.compute.manager [instance: 4af36ad5-7c64-4e0b-8160-919842a6c39c]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 663, in detach_volume

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1815792/+subscriptions