yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1582482] [NEW] After live-migration check source disk failed bdm.device_path lost

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: YaoZheng_ZTE <zheng.yao1@xxxxxxxxxx>
Date: Tue, 17 May 2016 03:09:26 -0000
Reply-to: Bug 1582482 <1582482@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description:
live-migration vm, the process raised exception in the function "check_can_live_migrate_source". Then, the vm looks like running and active. But, the bdm.connection_info lost 'device_path' key.
reproduce:
1. boot a vm from volume.
2. In the /nova/virt/libvirt/driver.py  function "check_can_live_migrate_source" structure throws an exception.
3. run nova live-migrate vm.
the nova-compute log as following:
2016-05-17 08:56:53.003 11926 ERROR oslo_messaging.rpc.dispatcher [req-a29e2ad4-8b28-4075-9c79-966086f99946 95d365ac3e3948c9be554a33855c6e07 853481fe4d1e4d1eb0136c7ecf46e5e7 - - -] Exception during message handling: 2C5_10_DELL05 is not on shared storage: Live migration can not be used without shared storage.
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8161, in check_can_live_migrate_source
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     dest_check_data)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     payload)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 405, in decorated_function
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6097, in check_can_live_migrate_source
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     block_device_info)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5699, in check_can_live_migrate_source
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     raise exception.InvalidSharedStorage(reason=reason, path=source)
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher InvalidSharedStorage: 2C5_10_DELL05 is not on shared storage: Live migration can not be used without shared storage.
2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher 
2016-05-17 08:56:53.006 11926 ERROR oslo_messaging._drivers.common [req-a29e2ad4-8b28-4075-9c79-966086f99946 95d365ac3e3948c9be554a33855c6e07 853481fe4d1e4d1eb0136c7ecf46e5e7 - - -] Returning exception 2C5_10_DELL05 is not on shared storage: Live migration can not be used without shared storage. to caller
4.check bdm information of the vm.
Before run live-migratin, the bdm infromation as following:
{"driver_volume_type": "iscsi", "connector": {"ip": "10.43.203.3", "host": "2C5_10_DELL05", "multipath": true, "initiator": "iqn.opencos.rh:995be1fac333"}, "serial": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "data": {"target_luns": [3, 3, 3, 3, 3, 3, 3], "target_iqns": ["iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91"], "device_path": "/dev/mapper/mpathdks", "target_discovered": false, "qos_specs": null, "target_iqn": "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "target_portals": ["172.168.102.25:3260", "172.168.103.25:3260", "172.168.104.25:3260", "172.168.101.25:3260", "172.168.102.27:3260", "172.168.103.27:3260", "172.168.104.27:3260"], "volume_id": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "target_lun": 3, "access_mode": "rw", "multipath_id": "mpathdks", "target_portal": "172.168.102.25:3260"}}

After run live-migration, the bdm information as following:
{"driver_volume_type": "iscsi", "connector": {"ip": "10.43.203.3", "host": "2C5_10_DELL05", "multipath": true, "initiator": "iqn.opencos.rh:995be1fac333"}, "serial": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "data": {"target_luns": [3, 3, 3, 3, 3, 3, 3], "target_iqns": ["iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91"], "target_discovered": false, "qos_specs": null, "target_iqn": "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "target_portals": ["172.168.102.25:3260", "172.168.103.25:3260", "172.168.104.25:3260", "172.168.101.25:3260", "172.168.102.27:3260", "172.168.103.27:3260", "172.168.104.27:3260"], "volume_id": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "target_lun": 3, "access_mode": "rw", "multipath_id": "mpathdks", "target_portal": "172.168.102.25:3260"}}
So, the device_path key was lost.

Environment:
I use mitika version.

** Affects: nova
     Importance: Undecided
     Assignee: YaoZheng_ZTE (zheng-yao1)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => YaoZheng_ZTE (zheng-yao1)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1582482

Title:
  After live-migration check source  disk failed bdm.device_path lost

Status in OpenStack Compute (nova):
  New

Bug description:
  Description:
  live-migration vm, the process raised exception in the function "check_can_live_migrate_source". Then, the vm looks like running and active. But, the bdm.connection_info lost 'device_path' key.
  reproduce:
  1. boot a vm from volume.
  2. In the /nova/virt/libvirt/driver.py  function "check_can_live_migrate_source" structure throws an exception.
  3. run nova live-migrate vm.
  the nova-compute log as following:
  2016-05-17 08:56:53.003 11926 ERROR oslo_messaging.rpc.dispatcher [req-a29e2ad4-8b28-4075-9c79-966086f99946 95d365ac3e3948c9be554a33855c6e07 853481fe4d1e4d1eb0136c7ecf46e5e7 - - -] Exception during message handling: 2C5_10_DELL05 is not on shared storage: Live migration can not be used without shared storage.
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     executor_callback)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 130, in _do_dispatch
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 8161, in check_can_live_migrate_source
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     dest_check_data)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     payload)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     return f(self, context, *args, **kw)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 405, in decorated_function
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     kwargs['instance'], e, sys.exc_info())
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     return function(self, context, *args, **kwargs)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6097, in check_can_live_migrate_source
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     block_device_info)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5699, in check_can_live_migrate_source
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher     raise exception.InvalidSharedStorage(reason=reason, path=source)
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher InvalidSharedStorage: 2C5_10_DELL05 is not on shared storage: Live migration can not be used without shared storage.
  2016-05-17 08:56:53.003 11926 TRACE oslo_messaging.rpc.dispatcher 
  2016-05-17 08:56:53.006 11926 ERROR oslo_messaging._drivers.common [req-a29e2ad4-8b28-4075-9c79-966086f99946 95d365ac3e3948c9be554a33855c6e07 853481fe4d1e4d1eb0136c7ecf46e5e7 - - -] Returning exception 2C5_10_DELL05 is not on shared storage: Live migration can not be used without shared storage. to caller
  4.check bdm information of the vm.
  Before run live-migratin, the bdm infromation as following:
  {"driver_volume_type": "iscsi", "connector": {"ip": "10.43.203.3", "host": "2C5_10_DELL05", "multipath": true, "initiator": "iqn.opencos.rh:995be1fac333"}, "serial": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "data": {"target_luns": [3, 3, 3, 3, 3, 3, 3], "target_iqns": ["iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91"], "device_path": "/dev/mapper/mpathdks", "target_discovered": false, "qos_specs": null, "target_iqn": "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "target_portals": ["172.168.102.25:3260", "172.168.103.25:3260", "172.168.104.25:3260", "172.168.101.25:3260", "172.168.102.27:3260", "172.168.103.27:3260", "172.168.104.27:3260"], "volume_id": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "target_lun": 3, "access_mode": "rw", "multipath_id": "mpathdks", "target_portal": "172.168.102.25:3260"}}

  After run live-migration, the bdm information as following:
  {"driver_volume_type": "iscsi", "connector": {"ip": "10.43.203.3", "host": "2C5_10_DELL05", "multipath": true, "initiator": "iqn.opencos.rh:995be1fac333"}, "serial": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "data": {"target_luns": [3, 3, 3, 3, 3, 3, 3], "target_iqns": ["iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91", "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:91"], "target_discovered": false, "qos_specs": null, "target_iqn": "iqn.2099-01.cn.com.zte:usp.spr11-4c:09:b4:b0:55:90", "target_portals": ["172.168.102.25:3260", "172.168.103.25:3260", "172.168.104.25:3260", "172.168.101.25:3260", "172.168.102.27:3260", "172.168.103.27:3260", "172.168.104.27:3260"], "volume_id": "78b80439-8bea-4900-aa7e-ddf2a9b0b8cf", "target_lun": 3, "access_mode": "rw", "multipath_id": "mpathdks", "target_portal": "172.168.102.25:3260"}}
  So, the device_path key was lost.

  Environment:
  I use mitika version.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1582482/+subscriptions
Follow ups

[Bug 1582482] Re: live-migration check source disk failed caused bdm.device_path lost
From: OpenStack Infra, 2016-06-27