yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1582482] Re: live-migration check source disk failed caused bdm.device_path lost

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1582482@xxxxxxxxxxxxxxxxxx>
Date: Mon, 27 Jun 2016 11:10:57 -0000
Reply-to: Bug 1582482 <1582482@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Reviewed:  https://review.openstack.org/330944
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c2f57a126f135879b96947a6734db9b0e01c2ad3
Submitter: Jenkins
Branch:    master

commit c2f57a126f135879b96947a6734db9b0e01c2ad3
Author: zhengyao1 <zheng.yao1@xxxxxxxxxx>
Date:   Fri Jun 17 17:42:00 2016 +0800

    live migration check source failed caused bdm.device_path lost
    
    Live migration vm, the process raised exception in the function
    "check_can_live_migrate_source". Then, the vm looks like running
    and active. But, the bdm.connection_info lost 'device_path' key.
    
    The reason is:the function check_can_live_migrate_source call
    the function _get_instance_block_device_info to check
    cinderclient, but this function will call cinder
    initialize_connection api interface and update
    BDM.connection_info, but cinder return connection_info has no
    'device_path' key. So after that, 'device_path' key lost.
    
    modification scheme:as for vm has been connected the volume,
    check in source host that is of small significance.
    Unfortunately, this interface will trigger the volume add SAN
    mapping group again.This interface call is very time consuming.
    In addition, For some SAN, this behavior may trigger errors. So
    this patch set the parameter 'refresh_conn_info' to false in
    order to remove the interface initialize_connection call.
    
    Change-Id: Ice8882b0c0e838072526ed285648bb8840968072
    Closes-Bug: #1582482


** Changed in: nova
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1582482

Title:
  live-migration check source  disk failed caused bdm.device_path lost

Status in OpenStack Compute (nova):
  Fix Released

Bug description:
  Description:
  live-migration vm, the process raised exception in the function "check_can_live_migrate_source". Then, the vm looks like running and active. But, the bdm.connection_info lost 'device_path' key.
  reproduce:
  1. boot a vm from volume.
  2. In the /nova/virt/libvirt/driver.py  function "check_can_live_migrate_source" structure throws an exception.
  3. run nova live-migrate vm.
  the nova-compute log as following:
  2016-05-17 08:56:53.003 11926 ERROR oslo_messaging.rpc.dispatcher [req-a29e2ad4-8b28-4075-9c79-966086f99946 95d365ac3e3948c9be554a33855c6e07 853481fe4d1e4d1eb0136c7ecf46e5e7 - - -] Exception during message handling: 2C5_10_DELL05 is not on shared storage: Live migration can not be used without shared storage.
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8161, in check_can_live_migrate_source
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     dest_check_data)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 405, in decorated_function
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6097, in check_can_live_migrate_source
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     block_device_info)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5699, in check_can_live_migrate_source
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     raise exception.InvalidSharedStorage(reason=reason, path=source)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher InvalidSharedStorage: 2C5_10_DELL05 is not on shared storage: Live migration can not be used without shared storage.
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher 
  2016-05-17 08:56:53.006 11926 ERROR oslo_messaging._drivers.common [req-a29e2ad4-8b28-4075-9c79-966086f99946 95d365ac3e3948c9be554a33855c6e07 853481fe4d1e4d1eb0136c7ecf46e5e7 - - -] Returning exception 2C5_10_DELL05 is not on shared storage: Live migration can not be used without shared storage. to caller
  4.check bdm information of the vm.
  Before run live-migratin, the bdm infromation as following:
  {"driver_volume_type": "iscsi", "connector": {"ip": "10.43.203.3", "host": "2C5_10_DELL05", "multipath": true, "initiator": "iqn.opencos.rh:995be1fac333"}, "serial": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "data": {"target_luns": [3, 3, 3, 3, 3, 3, 3], "target_iqns": ["iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91"], "device_path": "/dev/mapper/mpathdks", "target_discovered": false, "qos_specs": null, "target_iqn": "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "target_portals": ["172.168.102.25:3260", "172.168.103.25:3260", "172.168.104.25:3260", "172.168.101.25:3260", "172.168.102.27:3260", "172.168.103.27:3260", "172.168.104.27:3260"], "volume_id": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "target_lun": 3, "access_mode": "rw", "multipath_id": "mpathdks", "target_portal": "172.168.102.25:3260"}}

  After run live-migration, the bdm information as following:
  {"driver_volume_type": "iscsi", "connector": {"ip": "10.43.203.3", "host": "2C5_10_DELL05", "multipath": true, "initiator": "iqn.opencos.rh:995be1fac333"}, "serial": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "data": {"target_luns": [3, 3, 3, 3, 3, 3, 3], "target_iqns": ["iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91"], "target_discovered": false, "qos_specs": null, "target_iqn": "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "target_portals": ["172.168.102.25:3260", "172.168.103.25:3260", "172.168.104.25:3260", "172.168.101.25:3260", "172.168.102.27:3260", "172.168.103.27:3260", "172.168.104.27:3260"], "volume_id": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "target_lun": 3, "access_mode": "rw", "multipath_id": "mpathdks", "target_portal": "172.168.102.25:3260"}}
  So, the device_path key was lost.

  Environment:
  I use mitika version.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1582482/+subscriptions
References

[Bug 1582482] [NEW] After live-migration check source disk failed bdm.device_path lost
From: YaoZheng_ZTE, 2016-05-17