yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1406161] Re: boot from fcsan volume live migration caused bdm table lost connection_info value

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Doug Hellmann <doug@xxxxxxxxxxxxxxxx>
Date: Thu, 23 Jul 2015 14:34:33 -0000
Reply-to: Bug 1406161 <1406161@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: os-brick
       Status: Fix Committed => Fix Released

** Changed in: os-brick
    Milestone: None => 0.3.1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1406161

Title:
  boot from fcsan volume live migration caused bdm table lost
  connection_info value

Status in OpenStack Compute (nova):
  In Progress
Status in os-brick:
  Fix Released

Bug description:
  this issue reproduce steps:
  1. nova boot vm1 base fcsan volume, this instance bdm table connection_info as following:
  {"driver_volume_type": "fibre_channel", "serial": "63013f85-34fd-4040-a234-6ee57e1a8eeb", "data": {"device_path": "/dev/dm-11", "target_discovered": false, "devices": [{"device": "/dev/sdv", "host": "7", "id": "0", "channel": "0", "lun": "8"}, {"device": "/dev/sdx", "host": "8", "id": "0", "channel": "0", "lun": "8"}], "qos_specs": null, "volume_id": "63013f85-34fd-4040-a234-6ee57e1a8eeb", "target_lun": 8, "access_mode": "rw", "target_wwn": ["20014c09b4b04d0e", "20024c09b4b04d0e", "20034c09b4b04d0e", "20044c09b4b04d0e", "20014c09b4b04d18", "20024c09b4b04d18", "20034c09b4b04d18", "20044c09b4b04d18"], "multipath_id": "3500422790776c4d6"}}
  2. run nova live-migration vm1 fininshed, vm1 was migrated another host, but this instance bdm table connection_info as following:
  {"driver_volume_type": "fibre_channel", "serial": "63013f85-34fd-4040-a234-6ee57e1a8eeb", "data": { "target_discovered": false,  "qos_specs": null, "volume_id": "63013f85-34fd-4040-a234-6ee57e1a8eeb", "target_lun": 9, "access_mode": "rw", "target_wwn": ["20014c09b4b04d0e", "20024c09b4b04d0e", "20034c09b4b04d0e", "20044c09b4b04d0e", "20014c09b4b04d18", "20024c09b4b04d18", "20034c09b4b04d18", "20044c09b4b04d18"], "multipath_id": "3500422790776c4d6"}}

  we can find the key "device_path" and "devices"  was lost
  3.if we live-migration this vm1 again,will fail . Call chain are as follows:
  2014-12-16 05:19:12.039 9678 INFO nova.compute.manager [-] [instance: 5901a64a-32a6-448d-a535-5b03bb4df533] _post_live_migration() is started..
  2014-12-16 05:19:12.068 9678 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.43.177.244
  2014-12-16 05:19:12.139 9678 INFO nova.compute.manager [-] [instance: 5901a64a-32a6-448d-a535-5b03bb4df533] During sync_power_state the instance has a pending task. Skip.
  2014-12-16 05:19:12.512 9678 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     self.f(*self.args, **self.kw)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4987, in wait_for_live_migration
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     migrate_data)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     payload)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     six.reraise(self.type_, self.value, self.tb)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     return f(self, context, *args, **kw)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     e, sys.exc_info())
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     six.reraise(self.type_, self.value, self.tb)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 356, in decorated_function
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     return function(self, context, *args, **kwargs)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4824, in _post_live_migration
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     migrate_data)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5188, in post_live_migration
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     disk_dev)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1213, in volume_driver_method
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     return method(connection_info, *args, **kwargs)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     return f(*args, **kwargs)
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 1034, in disconnect_volume
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall     devicespath = connection_info['data']['device_path']
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall KeyError: 'device_path'
  2014-12-16 05:19:12.512 9678 TRACE nova.openstack.common.loopingcall 
  2014-12-16 05:19:31.629 9678 AUDIT nova.compute.resource_tracker [req-30cd2379-2ece-4577-a118-7b9daf4ef350 None None] Auditing locally available compute resources

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1406161/+subscriptions
References

[Bug 1406161] [NEW] boot from fcsan volume live migration caused bdm table lost connection_info value
From: YaoZheng_ZTE, 2014-12-29