yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1806711] [NEW] Message timeout leads to inconsistant disk attachments

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Gary Kotton <1806711@xxxxxxxxxxxxxxxxxx>
Date: Tue, 04 Dec 2018 15:16:32 -0000
Reply-to: Bug 1806711 <1806711@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

A message timeout causes a disk attachment to fail (which is legit). The problem is that the disk is actually attached to the VM and the rollback does not remove the disk from the VM and only sets the cinder/nova stats as available for the volume.
Subsequent attachments will fail.

Original stack is:

2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9] Traceback (most recent call last):
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 306, in attach
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     encryption=encryption)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 465, in detach_volume
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     if self._check_vio_import(instance):
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 458, in _check_vio_import
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     instance_meta = instance_metadata.InstanceMetadata(instance)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/api/metadata/base.py", line 141, in __init__
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     self.mappings = _format_instance_mapping(ctxt, instance)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/api/metadata/base.py", line 700, in _format_instance_mapping
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     ctxt, instance.uuid)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     args, kwargs)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 239, in object_class_action_versions
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     args=args, kwargs=kwargs)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     retry=self.retry)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     timeout=timeout, retry=retry)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in send
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     retry=retry)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 447, in _send
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     result = self._waiter.wait(msg_id, timeout)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in wait
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     message = self.waiters.get(msg_id, timeout=timeout)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     'to message ID %s' % msg_id)
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9] MessagingTimeout: Timed out waiting for a reply to message ID 1d358510329e41db90e87f6ff5176179
2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]


Subsequent attaches:
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     six.reraise(self.type_, self.value, self.tb)
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 273, in attach
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     device_type=self['device_type'], encryption=encryption)
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 454, in attach_volume
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return self._volumeops.attach_volume(connection_info, instance)
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 431, in attach_volume
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     self._attach_volume_vmdk(connection_info, instance, datastore)
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 389, in _attach_volume_vmdk
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     vmdk_path=vmdk.path, profile_id=profile_id)
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 112, in attach_disk_to_vm
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     vm_util.reconfigure_vm(self._session, vm_ref, vmdk_attach_config_spec)
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vm_util.py", line 2681, in reconfigure_vm
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     session._wait_for_task(reconfig_task)
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 720, in _wait_for_task
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return self.wait_for_task(task_ref)
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 396, in wait_for_task
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return evt.wait()
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return hubs.get_hub().switch()
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return self.greenlet.switch()
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     self.f(*self.args, **self.kw)
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 449, in _poll_task
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     raise task_ex
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9] VimFaultException: Failed to add disk 'scsi0:3'.
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9] Faults: ['GenericVmConfigFault']
2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]

** Affects: nova
     Importance: Undecided
     Assignee: Gary Kotton (garyk)
         Status: In Progress

** Changed in: nova
     Assignee: (unassigned) => Gary Kotton (garyk)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1806711

Title:
  Message timeout leads to inconsistant disk attachments

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  A message timeout causes a disk attachment to fail (which is legit). The problem is that the disk is actually attached to the VM and the rollback does not remove the disk from the VM and only sets the cinder/nova stats as available for the volume.
  Subsequent attachments will fail.

  Original stack is:

  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9] Traceback (most recent call last):
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 306, in attach
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     encryption=encryption)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 465, in detach_volume
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     if self._check_vio_import(instance):
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 458, in _check_vio_import
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     instance_meta = instance_metadata.InstanceMetadata(instance)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/api/metadata/base.py", line 141, in __init__
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     self.mappings = _format_instance_mapping(ctxt, instance)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/api/metadata/base.py", line 700, in _format_instance_mapping
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     ctxt, instance.uuid)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     args, kwargs)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 239, in object_class_action_versions
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     args=args, kwargs=kwargs)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     retry=self.retry)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in _send
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     timeout=timeout, retry=retry)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 458, in send
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     retry=retry)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 447, in _send
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     result = self._waiter.wait(msg_id, timeout)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in wait
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     message = self.waiters.get(msg_id, timeout=timeout)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 238, in get
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     'to message ID %s' % msg_id)
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9] MessagingTimeout: Timed out waiting for a reply to message ID 1d358510329e41db90e87f6ff5176179
  2018-12-03 04:30:58.003 2442 ERROR nova.virt.block_device [instance: 500f5906-1385-4afb-9317-bbab4599dad9]

  
  Subsequent attaches:
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     six.reraise(self.type_, self.value, self.tb)
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/block_device.py", line 273, in attach
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     device_type=self['device_type'], encryption=encryption)
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 454, in attach_volume
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return self._volumeops.attach_volume(connection_info, instance)
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 431, in attach_volume
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     self._attach_volume_vmdk(connection_info, instance, datastore)
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 389, in _attach_volume_vmdk
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     vmdk_path=vmdk.path, profile_id=profile_id)
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/volumeops.py", line 112, in attach_disk_to_vm
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     vm_util.reconfigure_vm(self._session, vm_ref, vmdk_attach_config_spec)
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/vm_util.py", line 2681, in reconfigure_vm
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     session._wait_for_task(reconfig_task)
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/nova/virt/vmwareapi/driver.py", line 720, in _wait_for_task
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return self.wait_for_task(task_ref)
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 396, in wait_for_task
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return evt.wait()
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return hubs.get_hub().switch()
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     return self.greenlet.switch()
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 76, in _inner
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     self.f(*self.args, **self.kw)
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]   File "/usr/lib/python2.7/dist-packages/oslo_vmware/api.py", line 449, in _poll_task
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]     raise task_ex
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9] VimFaultException: Failed to add disk 'scsi0:3'.
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9] Faults: ['GenericVmConfigFault']
  2018-12-03 08:09:13.528 5325 ERROR nova.compute.manager [instance: 500f5906-1385-4afb-9317-bbab4599dad9]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1806711/+subscriptions