yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1535373] [NEW] evacuation of volume backed instances fails due to Cinder being provided with connector information from the original host.

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lee Yarwood <lyarwood@xxxxxxxxxx>
Date: Mon, 18 Jan 2016 16:22:11 -0000
Reply-to: Bug 1535373 <1535373@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

1. Exact version of Nova/OpenStack you are running:
stable/kilo but this looks possible on stable/liberty and master still.

2. Relevant log files:

nova-compute.log

2016-01-16 05:57:09.522 18842 ERROR nova.compute.manager [req-0c36ed61-4f10-45c2-b0b5-342731fe79ba a290c48ff54d4a978474ec40401658ce 8d0f09d07ced49b8a1b7f991bfd9ba38 - - -] [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc] Setting instance vm_state to ERROR
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc] Traceback (most recent call last):
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6476, in _error_out_instance_on_exception
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     yield
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3113, in rebuild_instance
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     self._rebuild_default_impl(**kwargs)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2942, in _rebuild_default_impl
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     detach_block_devices(context, bdms)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3091, in detach_block_devices
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     destroy_bdm=False)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4956, in _detach_volume
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     self.volume_api.terminate_connection(context, volume_id, connector)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in wrapper
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     res = method(self, ctx, volume_id, *args, **kwargs)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 369, in terminate_connection
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     connector)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 448, in terminate_connection
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     {'connector': connector})
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 375, in _action
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     return self.api.client.post(url, body=body)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 118, in post
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     return self._cs_request(url, 'POST', **kwargs)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 112, in _cs_request
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     return self.request(url, method, **kwargs)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 105, in request
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     raise exceptions.from_response(resp, body)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-03984413-5ce6-413e-a792-fe0cc924e703)
2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]

cinder-volume.log

2016-01-16 05:57:09.513 12006 ERROR cinder.volume.manager [req-03984413-5ce6-413e-a792-fe0cc924e703 a290c48ff54d4a978474ec40401658ce 8d0f09d07ced49b8a1b7f991bfd9ba38 - - -] Unable to terminate volume connection: Failed to detach iSCSI target for volume 78dd98fd-33af-4714-927f-1a40c5e5f9b7.
2016-01-16 05:57:09.514 12006 ERROR oslo_messaging.rpc.dispatcher [req-03984413-5ce6-413e-a792-fe0cc924e703 a290c48ff54d4a978474ec40401658ce 8d0f09d07ced49b8a1b7f991bfd9ba38 - - -] Exception during message handling: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume 78dd98fd-33af-4714-927f-1a40c5e5f9b7.
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1181, in terminate_connection
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     raise exception.VolumeBackendAPIException(data=err_msg)
2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume 78dd98fd-33af-4714-927f-1a40c5e5f9b7.

The issue being that the initiator for the current host not existing on
the cinder-volume host as yet.

3. Reproduce steps:

- Using the LVM/iSCSI volume backend with Cinder.
- Boot an instance from a volume.
- Kill the host running the instance.
- Attempt to evacuate the instance from the host to a new host.

Expected result:
The instance is evacuated to a new host.

Actual result:
The instance fails to evacuate as Nova asks Cinder to terminate the connection of the volume using the connector information from the current host we are evacuating to and not the original host we are evacuating from.

** Affects: nova
     Importance: Undecided
     Assignee: Lee Yarwood (lyarwood)
         Status: In Progress

** Description changed:

  1. Exact version of Nova/OpenStack you are running:
  stable/kilo but this looks possible on stable/liberty and master still.
  
  2. Relevant log files:
  
  nova-compute.log
  
  2016-01-16 05:57:09.522 18842 ERROR nova.compute.manager [req-0c36ed61-4f10-45c2-b0b5-342731fe79ba a290c48ff54d4a978474ec40401658ce 8d0f09d07ced49b8a1b7f991bfd9ba38 - - -] [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc] Setting instance vm_state to ERROR
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc] Traceback (most recent call last):
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6476, in _error_out_instance_on_exception
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     yield
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3113, in rebuild_instance
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     self._rebuild_default_impl(**kwargs)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2942, in _rebuild_default_impl
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     detach_block_devices(context, bdms)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3091, in detach_block_devices
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     destroy_bdm=False)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4956, in _detach_volume
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     self.volume_api.terminate_connection(context, volume_id, connector)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in wrapper
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     res = method(self, ctx, volume_id, *args, **kwargs)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 369, in terminate_connection
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     connector)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 448, in terminate_connection
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     {'connector': connector})
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 375, in _action
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     return self.api.client.post(url, body=body)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 118, in post
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     return self._cs_request(url, 'POST', **kwargs)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 112, in _cs_request
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     return self.request(url, method, **kwargs)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 105, in request
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     raise exceptions.from_response(resp, body)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-03984413-5ce6-413e-a792-fe0cc924e703)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]
  
  cinder-volume.log
  
  2016-01-16 05:57:09.513 12006 ERROR cinder.volume.manager [req-03984413-5ce6-413e-a792-fe0cc924e703 a290c48ff54d4a978474ec40401658ce 8d0f09d07ced49b8a1b7f991bfd9ba38 - - -] Unable to terminate volume connection: Failed to detach iSCSI target for volume 78dd98fd-33af-4714-927f-1a40c5e5f9b7.
  2016-01-16 05:57:09.514 12006 ERROR oslo_messaging.rpc.dispatcher [req-03984413-5ce6-413e-a792-fe0cc924e703 a290c48ff54d4a978474ec40401658ce 8d0f09d07ced49b8a1b7f991bfd9ba38 - - -] Exception during message handling: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume 78dd98fd-33af-4714-927f-1a40c5e5f9b7.
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1181, in terminate_connection
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     raise exception.VolumeBackendAPIException(data=err_msg)
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume 78dd98fd-33af-4714-927f-1a40c5e5f9b7.
  
  The issue being that the initiator for the current host not existing on
  the cinder-volume host as yet.
  
  3. Reproduce steps:
  
  - Using the LVM/iSCSI volume backend with Cinder.
  - Boot an instance from a volume.
  - Kill the host running the instance.
  - Attempt to evacuate the instance from the host to a new host.
  
  Expected result:
  The instance is evacuated to a new host.
  
  Actual result:
- The instance fails to evacuate as Nova asks Cinder to terminate the connection of a volume using the connector information from the current host we are evacuating to and not the original host we are evacuating from.
+ The instance fails to evacuate as Nova asks Cinder to terminate the connection of the volume using the connector information from the current host we are evacuating to and not the original host we are evacuating from.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1535373

Title:
  evacuation of volume backed instances fails due to Cinder being
  provided with  connector information from the original host.

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  1. Exact version of Nova/OpenStack you are running:
  stable/kilo but this looks possible on stable/liberty and master still.

  2. Relevant log files:

  nova-compute.log

  2016-01-16 05:57:09.522 18842 ERROR nova.compute.manager [req-0c36ed61-4f10-45c2-b0b5-342731fe79ba a290c48ff54d4a978474ec40401658ce 8d0f09d07ced49b8a1b7f991bfd9ba38 - - -] [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc] Setting instance vm_state to ERROR
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc] Traceback (most recent call last):
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6476, in _error_out_instance_on_exception
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     yield
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3113, in rebuild_instance
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     self._rebuild_default_impl(**kwargs)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2942, in _rebuild_default_impl
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     detach_block_devices(context, bdms)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 3091, in detach_block_devices
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     destroy_bdm=False)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4956, in _detach_volume
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     self.volume_api.terminate_connection(context, volume_id, connector)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in wrapper
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     res = method(self, ctx, volume_id, *args, **kwargs)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 369, in terminate_connection
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     connector)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 448, in terminate_connection
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     {'connector': connector})
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/v2/volumes.py", line 375, in _action
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     return self.api.client.post(url, body=body)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 118, in post
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     return self._cs_request(url, 'POST', **kwargs)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 112, in _cs_request
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     return self.request(url, method, **kwargs)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]   File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 105, in request
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]     raise exceptions.from_response(resp, body)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-03984413-5ce6-413e-a792-fe0cc924e703)
  2016-01-16 05:57:09.522 18842 TRACE nova.compute.manager [instance: f7302235-20b6-4fb9-b129-aaad73a1b7dc]

  cinder-volume.log

  2016-01-16 05:57:09.513 12006 ERROR cinder.volume.manager [req-03984413-5ce6-413e-a792-fe0cc924e703 a290c48ff54d4a978474ec40401658ce 8d0f09d07ced49b8a1b7f991bfd9ba38 - - -] Unable to terminate volume connection: Failed to detach iSCSI target for volume 78dd98fd-33af-4714-927f-1a40c5e5f9b7.
  2016-01-16 05:57:09.514 12006 ERROR oslo_messaging.rpc.dispatcher [req-03984413-5ce6-413e-a792-fe0cc924e703 a290c48ff54d4a978474ec40401658ce 8d0f09d07ced49b8a1b7f991bfd9ba38 - - -] Exception during message handling: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume 78dd98fd-33af-4714-927f-1a40c5e5f9b7.
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1181, in terminate_connection
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher     raise exception.VolumeBackendAPIException(data=err_msg)
  2016-01-16 05:57:09.514 12006 TRACE oslo_messaging.rpc.dispatcher VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Unable to terminate volume connection: Failed to detach iSCSI target for volume 78dd98fd-33af-4714-927f-1a40c5e5f9b7.

  The issue being that the initiator for the current host not existing
  on the cinder-volume host as yet.

  3. Reproduce steps:

  - Using the LVM/iSCSI volume backend with Cinder.
  - Boot an instance from a volume.
  - Kill the host running the instance.
  - Attempt to evacuate the instance from the host to a new host.

  Expected result:
  The instance is evacuated to a new host.

  Actual result:
  The instance fails to evacuate as Nova asks Cinder to terminate the connection of the volume using the connector information from the current host we are evacuating to and not the original host we are evacuating from.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1535373/+subscriptions