yahoo-eng-team team mailing list archive

[Lee Yarwood <1886855@xxxxxxxxxxxxxxxxxx>]

** No longer affects: nova

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1886855

Title:
  Insufficient error handling when parsing iscsiadm -m node output with
  blank iscsi target

Status in Cinder:
  New
Status in os-brick:
  New

Bug description:
  We encountered the following error when attempting to reboot a VM with
  multiple attached volumes -

  2020-07-02 05:46:05.960 ERROR oslo_messaging.rpc.server [req-0c171deb-bc82-4687-91d6-76e8f95b8e19 **** service] Exception during message handling: IndexError: list index out of range
  ...
  2020-07-02 05:46:05.960 TRACE oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 157, in _get_iscsi_nodes
  2020-07-02 05:46:05.960 TRACE oslo_messaging.rpc.server lines.append((info[0].split(',')[0], info[1]))
  2020-07-02 05:46:05.960 TRACE oslo_messaging.rpc.server IndexError: list index out of range
  This is observed on os-brick version - 1.15.9

  The same code in current master branch -
  https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/iscsi.py#L136

  iscsiadm -m node output -
  172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-f1ff35f1-9716-4929-831f-32e7b207c742
  172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-5393e371-337f-4332-b39f-4926e4a1f9f7
  172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-1520a7d6-4351-416a-a703-c82f1bc9839d
  []:3260,-1 
  172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-fd632af2-45d9-4266-be67-b84e61fb3cbb
  172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-c1b325b9-7bd2-4d91-a3ef-295736e52eca
  172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-6a1a112e-1140-482d-9064-fe1b03391f2b

  The blank target causes an unhandled exception. A simple python code snippet to show the same - 
  >>> out = "172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-1520a7d6-4351-416a-a703-c82f1bc9839d\n[]:3260,-1\n172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-fd632af2-45d9-4266-be67-b84e61fb3cbb"
  >>> lines = []
  >>> out.splitlines()
  ['172.30.0.193:3260,-1 iqn.2010-10.org.openstack:volume-1520a7d6-4351-416a-a703-c82f1bc9839d', '[]:3260,-1', '172.30.0.191:3260,-1 iqn.2010-10.org.openstack:volume-fd632af2-45d9-4266-be67-b84e61fb3cbb']
  >>> for line in out.splitlines():
  ...     if line:
  ...             info = line.split()
  ...             lines.append((info[0].split(',')[0], info[1]))
  ...
  Traceback (most recent call last):
    File "<stdin>", line 4, in <module>

  The blank iscsi target was most probably due to corruption of the
  iscsi data during discovery.

  Using strace we could trace that blank target belongs to a OpenStack volume -
  open("/var/lib/iscsi/nodes/iqn.2010-10.org.openstack:volume-d88869e6-d27b-4121-9bd1-d8c86ce9d7e1/172.30.0.193,3260", O_RDONLY) = 5

  Expected:
  Management of that single volume associated with blank iscsi target to be affected.

  Observed:
  None of the volumes and volume backed VMs can be managed on the affected host.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1886855/+subscriptions