yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1425502] [NEW] VMware: Detaching volume fails with FileNotFoundException

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Vipin Balachandran <vbala@xxxxxxxxxx>
Date: Wed, 25 Feb 2015 12:39:54 -0000
Reply-to: Bug 1425502 <1425502@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Steps to reproduce:

a) Create nova instance vm-1 (assume that the datastore is a shared datastore 'ds-1' which is a member of datastore cluster dcls-1)
b) Create volume vol-1 with a storage profile which maps to ds-1
c) Attach vol-1 to vm-1
d) Migrate vm-1's datastore to shared datastore ds-2 which is a member of the dcls-1
(The vmdk of vol-1 *moves* to vm-1's new location after this step)
e) Detach vol-1


2015-02-25 18:05:00.014 ERROR oslo_messaging.rpc.dispatcher [req-124c1c0a-2a00-46b1-a790-0a2e5d6da871 admin demo] Exception during message handling: File /vmfs/volumes/54ae58ae-5db2e67f-38b1-0200066a0432/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19.vmdk was not found
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 416, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     payload)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 300, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     pass
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 285, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 328, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 316, in decorated_function
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4721, in detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self._detach_volume(context, instance, bdm)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4690, in _detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self.volume_api.roll_detaching(context, volume_id)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4678, in _detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     encryption=encryption)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 487, in detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     instance)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 539, in detach_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self._detach_volume_vmdk(connection_info, instance)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 504, in _detach_volume_vmdk
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     disk_type=vmdk.disk_type)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 455, in _consolidate_vmdk_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self._relocate_vmdk_volume(volume_ref, res_pool, datastore, host)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 384, in _relocate_vmdk_volume
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self._session._wait_for_task(task)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 673, in _wait_for_task
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return self.wait_for_task(task_ref)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 380, in wait_for_task
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return evt.wait()
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return hubs.get_hub().switch()
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return self.greenlet.switch()
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 76, in _inner
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self.f(*self.args, **self.kw)
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 417, in _poll_task
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     raise task_ex
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher FileNotFoundException: File /vmfs/volumes/54ae58ae-5db2e67f-38b1-0200066a0432/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19.vmdk was not found
2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher

** Affects: nova
     Importance: Undecided
     Assignee: Vipin Balachandran (vbala)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Vipin Balachandran (vbala)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1425502

Title:
  VMware: Detaching volume fails with FileNotFoundException

Status in OpenStack Compute (Nova):
  New

Bug description:
  Steps to reproduce:

  a) Create nova instance vm-1 (assume that the datastore is a shared datastore 'ds-1' which is a member of datastore cluster dcls-1)
  b) Create volume vol-1 with a storage profile which maps to ds-1
  c) Attach vol-1 to vm-1
  d) Migrate vm-1's datastore to shared datastore ds-2 which is a member of the dcls-1
  (The vmdk of vol-1 *moves* to vm-1's new location after this step)
  e) Detach vol-1

  
  2015-02-25 18:05:00.014 ERROR oslo_messaging.rpc.dispatcher [req-124c1c0a-2a00-46b1-a790-0a2e5d6da871 admin demo] Exception during message handling: File /vmfs/volumes/54ae58ae-5db2e67f-38b1-0200066a0432/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19.vmdk was not found
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 416, in decorated_function
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     payload)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 300, in decorated_function
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     pass
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 285, in decorated_function
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 328, in decorated_function
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 316, in decorated_function
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4721, in detach_volume
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self._detach_volume(context, instance, bdm)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4690, in _detach_volume
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self.volume_api.roll_detaching(context, volume_id)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/compute/manager.py", line 4678, in _detach_volume
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     encryption=encryption)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 487, in detach_volume
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     instance)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 539, in detach_volume
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self._detach_volume_vmdk(connection_info, instance)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 504, in _detach_volume_vmdk
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     disk_type=vmdk.disk_type)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 455, in _consolidate_vmdk_volume
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self._relocate_vmdk_volume(volume_ref, res_pool, datastore, host)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 384, in _relocate_vmdk_volume
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self._session._wait_for_task(task)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 673, in _wait_for_task
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return self.wait_for_task(task_ref)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 380, in wait_for_task
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return evt.wait()
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return hubs.get_hub().switch()
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     return self.greenlet.switch()
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/oslo.vmware/oslo_vmware/common/loopingcall.py", line 76, in _inner
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     self.f(*self.args, **self.kw)
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher   File "/opt/stack/oslo.vmware/oslo_vmware/api.py", line 417, in _poll_task
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher     raise task_ex
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher FileNotFoundException: File /vmfs/volumes/54ae58ae-5db2e67f-38b1-0200066a0432/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19/volume-dc966ab1-9d8b-4e3b-a3b4-7b610a9b0f19.vmdk was not found
  2015-02-25 18:05:00.014 TRACE oslo_messaging.rpc.dispatcher

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1425502/+subscriptions
Follow ups

[Bug 1425502] Re: VMware: Detaching volume fails with FileNotFoundException
From: Thierry Carrez, 2015-09-03
[Bug 1425502] [NEW] VMware: Detaching volume fails with FileNotFoundException
From: Vipin Balachandran, 2015-02-25
References

[Bug 1425502] [NEW] VMware: Detaching volume fails with FileNotFoundException
From: Vipin Balachandran, 2015-02-25