yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1423412] [NEW] Cannot boot instances from volumes in iSCSI multiple paths case

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Seiji Aguchi <seiji.aguchi.tr@xxxxxxxxxxx>
Date: Thu, 19 Feb 2015 01:44:27 -0000
Reply-to: Bug 1423412 <1423412@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

* Precondition:
- Nova: Add "iscsi_use_multipath = True" in [libvirt] section of nova.conf.
- Cinder: use iSCSI storage backend that supports multi-path.
  Nova-compute nodes and iSCSI ports of the storage array are connected to 2 LAN segments for mutli-pathing.
- One of the iSCSI paths is not active (for example, the cable is bad).

* How to reproduce:
Users create volume boot instance.
  nova boot --flavor m1.heat --boot-volume 24abb70b-c0ef-46c8-b734-6bc230fff1a7 test

* Actual results:
The instance fails to boot.
  nova list
  | 8df39f1b-d09d-42bf-8bf9-667c91a3ab34 | test  | ERROR  | -          | NOSTATE     | private=10.0.0.14 |

Error log in nova-compute is attached below.
It tries to issue "iscsiadm -m discovery -t sendtargets" command only on 1 path and gives up with ProcessExecutionError.

* Expected results:
The instance successfully boots using another iSCSI path.
It is desirable that users can use the cloud system normally, even while the administrators is resolving the path failure.


* Nova-compute Log:

2015-02-12 15:43:57.439 DEBUG oslo_concurrency.processutils [-] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.2:3260" returned: 4 in 95 pid=9253) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:216
2015-02-12 15:43:57.439 DEBUG oslo_concurrency.processutils [-] u'sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.2:3260' failed. Not Retrying) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:249
2015-02-12 15:43:57.439 DEBUG oslo_concurrency.lockutils [-] Lock "connect_volume" released by "connect_volume" :: held 95.187s from (pid=9253) inner /usr/lib/python2.7/site-packages/oslo_lockutils.py:442
2015-02-12 15:43:57.439 ERROR nova.compute.manager [-] [instance: 98796b31-8750-469d-87b1-537489b6749e] Instance failed to spawn
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Traceback (most recent call last):
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/compute/manager.py", line 2318, in _build_resources
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     yield resources
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/compute/manager.py", line 2188, in _build_and_run_instance
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     flavor=flavor)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2353, in spawn
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     flavor=flavor)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4165, in _get_guest_xml
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     context, flavor=flavor)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4028, in _get_guest_config
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     flavor, guest.os_type)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3213, in _get_guest_storage_c
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     self._connect_volume(connection_info, info)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 977, in _connect_volume
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     driver.connect_volume(connection_info, disk_info)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 431, in in
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     return f(*args, **kwargs)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/volume.py", line 362, in connect_volume
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     out = self._run_iscsiadm_discover(iscsi_properties)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/volume.py", line 498, in _run_iscsiadm_discove
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     check_exit_code=[0, 255])[0] or ""
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/volume.py", line 768, in _run_iscsiadm_bare
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     check_exit_code=check_exit_code)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/utils.py", line 197, in execute
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     return processutils.execute(*cmd, **kwargs)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     cmd=sanitized_cmd)
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] ProcessExecutionError: Unexpected error while running command.
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.2:3260
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Exit code: 4
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Stdout: u''
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Stderr: u'iscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm login retries (reopen_max) 5 exceeded\niscsiadm: Could not perform SendTargets discovery: encountered connection failure\n'
2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1423412

Title:
  Cannot boot instances from volumes in iSCSI multiple paths case

Status in OpenStack Compute (Nova):
  New

Bug description:
  * Precondition:
  - Nova: Add "iscsi_use_multipath = True" in [libvirt] section of nova.conf.
  - Cinder: use iSCSI storage backend that supports multi-path.
    Nova-compute nodes and iSCSI ports of the storage array are connected to 2 LAN segments for mutli-pathing.
  - One of the iSCSI paths is not active (for example, the cable is bad).

  * How to reproduce:
  Users create volume boot instance.
    nova boot --flavor m1.heat --boot-volume 24abb70b-c0ef-46c8-b734-6bc230fff1a7 test

  * Actual results:
  The instance fails to boot.
    nova list
    | 8df39f1b-d09d-42bf-8bf9-667c91a3ab34 | test  | ERROR  | -          | NOSTATE     | private=10.0.0.14 |

  Error log in nova-compute is attached below.
  It tries to issue "iscsiadm -m discovery -t sendtargets" command only on 1 path and gives up with ProcessExecutionError.

  * Expected results:
  The instance successfully boots using another iSCSI path.
  It is desirable that users can use the cloud system normally, even while the administrators is resolving the path failure.

  
  * Nova-compute Log:

  2015-02-12 15:43:57.439 DEBUG oslo_concurrency.processutils [-] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.2:3260" returned: 4 in 95 pid=9253) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:216
  2015-02-12 15:43:57.439 DEBUG oslo_concurrency.processutils [-] u'sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.2:3260' failed. Not Retrying) execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:249
  2015-02-12 15:43:57.439 DEBUG oslo_concurrency.lockutils [-] Lock "connect_volume" released by "connect_volume" :: held 95.187s from (pid=9253) inner /usr/lib/python2.7/site-packages/oslo_lockutils.py:442
  2015-02-12 15:43:57.439 ERROR nova.compute.manager [-] [instance: 98796b31-8750-469d-87b1-537489b6749e] Instance failed to spawn
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Traceback (most recent call last):
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/compute/manager.py", line 2318, in _build_resources
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     yield resources
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/compute/manager.py", line 2188, in _build_and_run_instance
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     flavor=flavor)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2353, in spawn
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     flavor=flavor)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4165, in _get_guest_xml
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     context, flavor=flavor)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4028, in _get_guest_config
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     flavor, guest.os_type)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3213, in _get_guest_storage_c
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     self._connect_volume(connection_info, info)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 977, in _connect_volume
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     driver.connect_volume(connection_info, disk_info)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 431, in in
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     return f(*args, **kwargs)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/volume.py", line 362, in connect_volume
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     out = self._run_iscsiadm_discover(iscsi_properties)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/volume.py", line 498, in _run_iscsiadm_discove
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     check_exit_code=[0, 255])[0] or ""
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/virt/libvirt/volume.py", line 768, in _run_iscsiadm_bare
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     check_exit_code=check_exit_code)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/opt/stack/nova/nova/utils.py", line 197, in execute
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     return processutils.execute(*cmd, **kwargs)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]   File "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py", line 224, in
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]     cmd=sanitized_cmd)
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] ProcessExecutionError: Unexpected error while running command.
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf iscsiadm -m discovery -t sendtargets -p 192.168.1.2:3260
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Exit code: 4
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Stdout: u''
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e] Stderr: u'iscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm: connect to 192.168.1.2 timed out\niscsiadm login retries (reopen_max) 5 exceeded\niscsiadm: Could not perform SendTargets discovery: encountered connection failure\n'
  2015-02-12 15:43:57.439 TRACE nova.compute.manager [instance: 98796b31-8750-469d-87b1-537489b6749e]

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1423412/+subscriptions
Follow ups

[Bug 1423412] [NEW] Cannot boot instances from volumes in iSCSI multiple paths case
From: Seiji Aguchi, 2015-02-19
References

[Bug 1423412] [NEW] Cannot boot instances from volumes in iSCSI multiple paths case
From: Seiji Aguchi, 2015-02-19