yahoo-eng-team team mailing list archive

[Chuck Short <chuck.short@xxxxxxxxxxxxx>]

** Changed in: nova/kilo
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1423772

Title:
  During live-migration Nova expects identical IQN from attached
  volume(s)

Status in OpenStack Compute (nova):
  Fix Released
Status in OpenStack Compute (nova) kilo series:
  Fix Released

Bug description:
  When attempting to do a live-migration on an instance with one or more
  attached volumes, Nova expects that the IQN will be exactly the same
  as it's attaching the volume(s) to the new host. This conflicts with
  the Cinder settings such as "hp3par_iscsi_ips" which allows for
  multiple IPs for the purpose of load balancing.

  Example:
  An instance on Host A has a volume attached at "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
  An attempt is made to migrate the instance to Host B.
  Cinder sends the request to attach the volume to the new host.
  Cinder gives the new host "/dev/disk/by-path/ip-10.10.120.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"
  Nova looks for the volume on the new host at the old location "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2"

  The following error appears in n-cpu in this case:

  2015-02-19 17:09:05.574 ERROR nova.virt.libvirt.driver [-] [instance: b6fa616f-4e78-42b1-a747-9d081a4701df] Live Migration failure: Failed to open file '/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2': No such file or directory
  Traceback (most recent call last):
    File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/poll.py", line 115, in wait
      listener.cb(fileno)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 212, in main
      result = function(*args, **kwargs)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5426, in _live_migration
      recover_method(context, instance, dest, block_migration)
    File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5393, in _live_migration
      CONF.libvirt.live_migration_bandwidth)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 183, in doit
      result = proxy_call(self._autowrap, f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 141, in proxy_call
      rv = execute(f, *args, **kwargs)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 122, in execute
      six.reraise(c, e, tb)
    File "/usr/local/lib/python2.7/dist-packages/eventlet/tpool.py", line 80, in tworker
      rv = meth(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1582, in migrateToURI2
      if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self)
  libvirtError: Failed to open file '/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2': No such file or directory
  Removing descriptor: 3

  
  When looking at the nova DB, this is the state of block_device_mapping prior to the migration attempt:

  mysql> select * from block_device_mapping where instance_uuid='b6fa616f-4e78-42b1-a747-9d081a4701df' and deleted=0;
  +---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
  | created_at          | updated_at          | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id                            | volume_size | no_device | connection_info                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   | instance_uuid                        | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id |
  +---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
  | 2015-02-20 00:57:01 | 2015-02-20 01:03:06 | NULL       |  3 | /dev/vda    |                     0 | NULL        | e031b804-f824-45f1-a9fa-b9330ba061e0 |        NULL |      NULL | {"driver_volume_type": "iscsi", "serial": "e031b804-f824-45f1-a9fa-b9330ba061e0", "data": {"host_device": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-1", "target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 1, "access_mode": "rw"}}                                                                                                                 | b6fa616f-4e78-42b1-a747-9d081a4701df |       0 | volume      | volume           | NULL         | disk        | virtio   |          0 | NULL     |
  | 2015-02-20 01:07:11 | 2015-02-20 01:07:19 | NULL       |  5 | /dev/vdb    |                     0 | NULL        | c3009a3d-549d-4ee5-b7a6-b0eac6382beb |        NULL |      NULL | {"driver_volume_type": "iscsi", "serial": "c3009a3d-549d-4ee5-b7a6-b0eac6382beb", "data": {"device_path": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2", "host_device": "/dev/disk/by-path/ip-10.10.220.244:3260-iscsi-iqn.2000-05.com.3pardata:22210002ac002a13-lun-2", "target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 2, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df |       0 | volume      | volume           | NULL         | NULL        | NULL     |       NULL | NULL     |
  +---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
  2 rows in set (0.00 sec)



  
  Then after the error message comes up in n-cpu:

  +---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
  | created_at          | updated_at          | deleted_at | id | device_name | delete_on_termination | snapshot_id | volume_id                            | volume_size | no_device | connection_info                                                                                                                                                                                                                                                                   | instance_uuid                        | deleted | source_type | destination_type | guest_format | device_type | disk_bus | boot_index | image_id |
  +---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
  | 2015-02-20 00:57:01 | 2015-02-20 01:08:55 | NULL       |  3 | /dev/vda    |                     0 | NULL        | e031b804-f824-45f1-a9fa-b9330ba061e0 |        NULL |      NULL | {"driver_volume_type": "iscsi", "serial": "e031b804-f824-45f1-a9fa-b9330ba061e0", "data": {"target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 0, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df |       0 | volume      | volume           | NULL         | disk        | virtio   |          0 | NULL     |
  | 2015-02-20 01:07:11 | 2015-02-20 01:08:57 | NULL       |  5 | /dev/vdb    |                     0 | NULL        | c3009a3d-549d-4ee5-b7a6-b0eac6382beb |        NULL |      NULL | {"driver_volume_type": "iscsi", "serial": "c3009a3d-549d-4ee5-b7a6-b0eac6382beb", "data": {"target_discovered": true, "qos_specs": null, "target_iqn": "iqn.2000-05.com.3pardata:22210002ac002a13", "target_portal": "10.10.220.244:3260", "target_lun": 1, "access_mode": "rw"}} | b6fa616f-4e78-42b1-a747-9d081a4701df |       0 | volume      | volume           | NULL         | NULL        | NULL     |       NULL | NULL     |
  +---------------------+---------------------+------------+----+-------------+-----------------------+-------------+--------------------------------------+-------------+-----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------------------------------+---------+-------------+------------------+--------------+-------------+----------+------------+----------+
  2 rows in set (0.00 sec)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1423772/+subscriptions