yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1593742] [NEW] Detach VMDK volume fails if shadow VM unregistered and registered back

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Chinmaya Bharadwaj <chinmaya-bharadwaj.a@xxxxxx>
Date: Fri, 17 Jun 2016 14:00:18 -0000
Reply-to: Bug 1593742 <1593742@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
When trying nova VMDK volume detach from an instance hosted on vCenter, the detach fail if the shadow VM used for vmdk volume is unregistered and registered back in vCenter.


Steps to reproduce
==================

* Attach a VMDK volume to an instance
* Unregister/delete the shadow VM. [A VM just contains the VMDK matching the cinder UUID]
* register/create the VM back with the same name
* Try detach 


Expected result
===============
It should detach the volume from the instance

Actual result
=============

It fails to detach.

Because in the vCenter, due to un-registering the vm MOid changed. In block_device_mapping table we store
this information (connection_info) and use it to form the volume ref during detach in the driver. 

https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/volumeops.py#L305

I think we should not be relying on that, instead make the vCenter call
and get the MOID by name.

Logs
====
Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
Faults: [ManagedObjectNotFound]
Details: {'obj': 'vm-1031'}
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     payload)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 387, in decorated_function
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 375, in decorated_function
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4892, in detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     attachment_id=attachment_id)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4835, in _detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     connection_info = self._driver_detach_volume(context, instance, bdm)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4787, in _driver_detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self.volume_api.roll_detaching(context, volume_id)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4775, in _driver_detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     encryption=encryption)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 391, in detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return self._volumeops.detach_volume(connection_info, instance)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 582, in detach_volume
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self._detach_volume_vmdk(connection_info, instance)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 530, in _detach_volume_vmdk
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     vmdk = "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 635, in get_vmdk_info
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     "config.hardware.device")
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 596, in _call_method
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return self.invoke_api(module, method, self.vim, *args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 348, in invoke_api
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return _invoke_api(module, method, *args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 123, in func
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return evt.wait()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return hubs.get_hub().switch()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return self.greenlet.switch()
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 124, in _inner
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     idle = self.f(*self.args, **self.kw)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 96, in _func
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     result = f(*args, **kwargs)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 332, in _invoke_api
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     details=excep.details)
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher ManagedObjectNotFoundException: The object has already been deleted or has not been completely created
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Faults: [ManagedObjectNotFound]
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Details: {'obj': 'vm-1031'}
2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher


Environment
===========

nova driver: VMWareVCDriver
cinder driver: VMDkdriver
Observing the issue on "stable/mitaka". I believe this issue persists in master as well.

** Affects: nova
     Importance: Undecided
     Assignee: Chinmaya Bharadwaj (acbharadwaj)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Chinmaya Bharadwaj (acbharadwaj)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1593742

Title:
  Detach VMDK volume fails if shadow VM unregistered and registered back

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  When trying nova VMDK volume detach from an instance hosted on vCenter, the detach fail if the shadow VM used for vmdk volume is unregistered and registered back in vCenter.

  
  Steps to reproduce
  ==================

  * Attach a VMDK volume to an instance
  * Unregister/delete the shadow VM. [A VM just contains the VMDK matching the cinder UUID]
  * register/create the VM back with the same name
  * Try detach 

  
  Expected result
  ===============
  It should detach the volume from the instance

  Actual result
  =============

  It fails to detach.

  Because in the vCenter, due to un-registering the vm MOid changed. In block_device_mapping table we store
  this information (connection_info) and use it to form the volume ref during detach in the driver. 

  https://github.com/openstack/nova/blob/master/nova/virt/vmwareapi/volumeops.py#L305

  I think we should not be relying on that, instead make the vCenter
  call and get the MOID by name.

  Logs
  ====
  Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
  Faults: [ManagedObjectNotFound]
  Details: {'obj': 'vm-1031'}
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     incoming.message))
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 110, in wrapped
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     payload)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 387, in decorated_function
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 375, in decorated_function
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4892, in detach_volume
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     attachment_id=attachment_id)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4835, in _detach_volume
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     connection_info = self._driver_detach_volume(context, instance, bdm)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4787, in _driver_detach_volume
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self.volume_api.roll_detaching(context, volume_id)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 221, in __exit__
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self.force_reraise()
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 197, in force_reraise
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4775, in _driver_detach_volume
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     encryption=encryption)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 391, in detach_volume
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return self._volumeops.detach_volume(connection_info, instance)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 582, in detach_volume
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     self._detach_volume_vmdk(connection_info, instance)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 530, in _detach_volume_vmdk
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     vmdk = "/opt/stack/nova/nova/virt/vmwareapi/vm_util.py", line 635, in get_vmdk_info
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     "config.hardware.device")
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 596, in _call_method
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return self.invoke_api(module, method, self.vim, *args, **kwargs)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 348, in invoke_api
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return _invoke_api(module, method, *args, **kwargs)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 123, in func
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return evt.wait()
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return hubs.get_hub().switch()
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     return self.greenlet.switch()
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/common/loopingcall.py", line 124, in _inner
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     idle = self.f(*self.args, **self.kw)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 96, in _func
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     result = f(*args, **kwargs)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_vmware/api.py", line 332, in _invoke_api
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher     details=excep.details)
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher ManagedObjectNotFoundException: The object has already been deleted or has not been completely created
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Cause: Server raised fault: 'The object has already been deleted or has not been completely created'
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Faults: [ManagedObjectNotFound]
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher Details: {'obj': 'vm-1031'}
  2014-11-19 06:05:55.086 29958 ERROR oslo_messaging.rpc.dispatcher

  
  Environment
  ===========

  nova driver: VMWareVCDriver
  cinder driver: VMDkdriver
  Observing the issue on "stable/mitaka". I believe this issue persists in master as well.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1593742/+subscriptions