yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #28637
[Bug 1423772] [NEW] During live-migration Nova expects identical IQN from attached volume(s)
Public bug reported:
When attempting to do a live-migration on an instance with one or more
attached volumes, Nova expects that the IQN will be exactly the same as
it's attaching the volume(s) to the new host. This conflicts with the
Cinder settings such as "hp3par_iscsi_ips" which allows for multiple IPs
for the purpose of load balancing.
Example:
An instance on Host A has a volume attached at "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
An attempt is made to migrate the instance to Host B.
Cinder sends the request to attach the volume to the new host.
Cinder gives the new host "/dev/disk/by-path/ip-10.10.120.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
Nova looks for the volume on the new host at the old location "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
The following error appears in n-cpu in this case:
2015-02-19 17:09:05.574 ERROR nova.virt.libvirt.driver [-] [instance: b6fa616f-4e78-42b1-a747-9d081a4701df] Live Migration failure: Failed to open file '/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2': No such file or directory
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
listener.cb(fileno)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
result = function(*args, **kwargs)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5426, in _live_migration
recover_method(context, instance, dest, block_migration)
File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5393, in _live_migration
CONF.libvirt.live_migration_bandwidth)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
six.reraise(c, e, tb)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
rv = meth(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1582, in migrateToURI2
if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: Failed to open file '/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2': No such file or directory
Removing descriptor: 3
When looking at the nova DB, this is the state of block_device_mapping prior to the migration attempt:
mysql> select * from block_device_mapping where instance_uuid='b6fa616f-4e78-42b1-a747-9d081a4701df' and deleted=0;
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| created_at | updated_at | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id | volume_size | no_device | connection_info | instance_uuid | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| 2015-02-20 00:57:01 | 2015-02-20 01:03:06 | NULL | 3 | /dev/vda | 0 | NULL | e031b804-f824-45f1-a9fa-b9330ba061e0 | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "e031b804-f824-45f1-a9fa-b9330ba061e0", "data": {"host_device": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-1", "target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 1, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | disk | virtio | 0 | NULL |
| 2015-02-20 01:07:11 | 2015-02-20 01:07:19 | NULL | 5 | /dev/vdb | 0 | NULL | c3009a3d-549d-4ee5-b7a6-b0eac6382beb | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "c3009a3d-549d-4ee5-b7a6-b0eac6382beb", "data": {"device_path": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2", "host_device": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2", "target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 2, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | NULL | NULL | NULL | NULL |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
2 rows in set (0.00 sec)
Then after the error message comes up in n-cpu:
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| created_at | updated_at | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id | volume_size | no_device | connection_info | instance_uuid | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| 2015-02-20 00:57:01 | 2015-02-20 01:08:55 | NULL | 3 | /dev/vda | 0 | NULL | e031b804-f824-45f1-a9fa-b9330ba061e0 | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "e031b804-f824-45f1-a9fa-b9330ba061e0", "data": {"target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 0, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | disk | virtio | 0 | NULL |
| 2015-02-20 01:07:11 | 2015-02-20 01:08:57 | NULL | 5 | /dev/vdb | 0 | NULL | c3009a3d-549d-4ee5-b7a6-b0eac6382beb | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "c3009a3d-549d-4ee5-b7a6-b0eac6382beb", "data": {"target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 1, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | NULL | NULL | NULL | NULL |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
2 rows in set (0.00 sec)
** Affects: nova
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1423772
Title:
During live-migration Nova expects identical IQN from attached
volume(s)
Status in OpenStack Compute (Nova):
New
Bug description:
When attempting to do a live-migration on an instance with one or more
attached volumes, Nova expects that the IQN will be exactly the same
as it's attaching the volume(s) to the new host. This conflicts with
the Cinder settings such as "hp3par_iscsi_ips" which allows for
multiple IPs for the purpose of load balancing.
Example:
An instance on Host A has a volume attached at "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
An attempt is made to migrate the instance to Host B.
Cinder sends the request to attach the volume to the new host.
Cinder gives the new host "/dev/disk/by-path/ip-10.10.120.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
Nova looks for the volume on the new host at the old location "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
The following error appears in n-cpu in this case:
2015-02-19 17:09:05.574 ERROR nova.virt.libvirt.driver [-] [instance: b6fa616f-4e78-42b1-a747-9d081a4701df] Live Migration failure: Failed to open file '/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2': No such file or directory
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
listener.cb(fileno)
File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
result = function(*args, **kwargs)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5426, in _live_migration
recover_method(context, instance, dest, block_migration)
File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5393, in _live_migration
CONF.libvirt.live_migration_bandwidth)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
result = proxy_call(self._autowrap, f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
rv = execute(f, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
six.reraise(c, e, tb)
File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
rv = meth(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1582, in migrateToURI2
if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
libvirtError: Failed to open file '/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2': No such file or directory
Removing descriptor: 3
When looking at the nova DB, this is the state of block_device_mapping prior to the migration attempt:
mysql> select * from block_device_mapping where instance_uuid='b6fa616f-4e78-42b1-a747-9d081a4701df' and deleted=0;
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| created_at | updated_at | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id | volume_size | no_device | connection_info | instance_uuid | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| 2015-02-20 00:57:01 | 2015-02-20 01:03:06 | NULL | 3 | /dev/vda | 0 | NULL | e031b804-f824-45f1-a9fa-b9330ba061e0 | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "e031b804-f824-45f1-a9fa-b9330ba061e0", "data": {"host_device": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-1", "target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 1, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | disk | virtio | 0 | NULL |
| 2015-02-20 01:07:11 | 2015-02-20 01:07:19 | NULL | 5 | /dev/vdb | 0 | NULL | c3009a3d-549d-4ee5-b7a6-b0eac6382beb | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "c3009a3d-549d-4ee5-b7a6-b0eac6382beb", "data": {"device_path": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2", "host_device": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2", "target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 2, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | NULL | NULL | NULL | NULL |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
2 rows in set (0.00 sec)
Then after the error message comes up in n-cpu:
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| created_at | updated_at | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id | volume_size | no_device | connection_info | instance_uuid | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
| 2015-02-20 00:57:01 | 2015-02-20 01:08:55 | NULL | 3 | /dev/vda | 0 | NULL | e031b804-f824-45f1-a9fa-b9330ba061e0 | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "e031b804-f824-45f1-a9fa-b9330ba061e0", "data": {"target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 0, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | disk | virtio | 0 | NULL |
| 2015-02-20 01:07:11 | 2015-02-20 01:08:57 | NULL | 5 | /dev/vdb | 0 | NULL | c3009a3d-549d-4ee5-b7a6-b0eac6382beb | NULL | NULL | {"driver_volume_type": "iscsi", "serial": "c3009a3d-549d-4ee5-b7a6-b0eac6382beb", "data": {"target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 1, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df | 0 | volume | volume | NULL | NULL | NULL | NULL | NULL |
+---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
2 rows in set (0.00 sec)
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1423772/+subscriptions
Follow ups
References
