yahoo-eng-team team mailing list archive

[Sampath Priyankara <sampath.priyankara@xxxxxxxxxxxxx>]

Public bug reported:

iSCSI volume detach does not correctly remove the multipath device
descriptors

tested environment:
nova-compute on Ubuntu 14.04.1, iscsi_use_multipath=True and iSCSI volume backend is EMC VNX 5300.

 I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully.  3rd volume also successfully detached but ended up with  failed multipaths. 
Here is the terminal log for last volume detach.

openstack@W1DEV103:~/devstack$ cinder list
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
|
 ID
 | Status | Name | Size | Volume Type | Bootable |
 Attached to
 |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | in-use | None | 1 |
 None
 | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e |
+--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l
Fri Sep 19 21:38:13 JST 2014
360060160cf0036002d1475f6e73fe411 dm-2 DGC,VRAID
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- 4:0:0:42 sdb 8:16 active undef running
| |- 5:0:0:42 sdd 8:48 active undef running
| |- 6:0:0:42 sdf 8:80 active undef running
| `- 7:0:0:42 sdh 8:112 active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- 11:0:0:42 sdp 8:240 active undef running
|- 8:0:0:42 sdj 8:144 active undef running
|- 9:0:0:42 sdl 8:176 active undef running
`- 10:0:0:42 sdn 8:208 active undef running
openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
Fri Sep 19 21:38:19 JST 2014
tcp: [10] 172.23.58.228:3260,4 iqn.1992-04.com.emc:cx.fcn00133400150.a7
tcp: [3] 172.23.58.238:3260,8 iqn.1992-04.com.emc:cx.fcn00133400150.b7
tcp: [4] 172.23.58.235:3260,20 iqn.1992-04.com.emc:cx.fcn00133400150.b4
tcp: [5] 172.23.58.236:3260,6 iqn.1992-04.com.emc:cx.fcn00133400150.b5
tcp: [6] 172.23.58.237:3260,19 iqn.1992-04.com.emc:cx.fcn00133400150.b6
tcp: [7] 172.23.58.225:3260,16 iqn.1992-04.com.emc:cx.fcn00133400150.a4
tcp: [8] 172.23.58.226:3260,2 iqn.1992-04.com.emc:cx.fcn00133400150.a5
tcp: [9] 172.23.58.227:3260,17 iqn.1992-04.com.emc:cx.fcn00133400150.a6

openstack@W1DEV103:~/devstack$ nova volume-detach 5bd68785-4acf-43ab-ae13-11b1edc3a62e
56a63288-5cc0-4f5c-9197-cde731172dd8
openstack@W1DEV103:~/devstack$
openstack@W1DEV103:~/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
|
 ID
 | Status | Name | Size | Volume Type | Bootable |
 Attached to
 |
+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | detaching | None | 1 |
 None
 | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e|

+--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
openstack@W1DEV103:~/devstack$
openstack@W1DEV103:~/devstack$ cinder list
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|
 ID
 | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
| 56a63288-5cc0-4f5c-9197-cde731172dd8 | available | None | 1 |
 None
 | false |
+--------------------------------------+-----------+------+------+-------------+----------+-------------+
|
openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l
Fri Sep 19 21:39:23 JST 2014
360060160cf0036002d1475f6e73fe411 dm-2 ,
size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=-1 status=active
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| |- #:#:#:# - #:# active undef running
| `- #:#:#:# - #:# active undef running
`-+- policy='round-robin 0' prio=-1 status=enabled
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
|- #:#:#:# - #:# active undef running
`- #:#:#:# - #:# active undef running
openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
Fri Sep 19 21:39:27 JST 2014
iscsiadm: No active sessions.

Then I manually removed the multipaths,
openstack@W1CN103:/etc/iscsi$ sudo multipath -f 360060160cf0036002d1475f6e73fe411
openstack@W1CN103:/etc/iscsi$ sudo multipath -l
openstack@W1CN103:/etc/iscsi$

 I think the problem is in,
virt/libvirt/volume.py:LibvirtISCSIVolumeDriver
 def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device):

End of this method executes following code to call remove_multipath_device_descriptor for remove the multipaths by multipath -f before return.
        self._remove_multipath_device_descriptor(multipath_device)
        return
Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one),  this method return without calling _remove_multipath_device_descriptor due to following if statement. 
 
        if not in_use:
            # disconnect if no other multipath devices with same iqn
            self._disconnect_mpath(iscsi_properties, ips_iqns)
            return
It just disconnect them but not remove them.
One of the reasons why we have to remove them is,
https://bugs.launchpad.net/nova/+bug/1223975

IMO, we should call _remove_multipath_device_descriptor in above if
statement before return.

** Affects: nova
     Importance: Undecided
     Assignee: Sampath Priyankara (sampath-priyankara)
         Status: New

** Changed in: nova
     Assignee: (unassigned) => Sampath Priyankara (sampath-priyankara)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1374999

Title:
  iSCSI volume detach does not correctly remove the multipath device
  descriptors

Status in OpenStack Compute (Nova):
  New

Bug description:
  iSCSI volume detach does not correctly remove the multipath device
  descriptors

  tested environment:
  nova-compute on Ubuntu 14.04.1, iscsi_use_multipath=True and iSCSI volume backend is EMC VNX 5300.

   I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully.  3rd volume also successfully detached but ended up with  failed multipaths. 
  Here is the terminal log for last volume detach.

  openstack@W1DEV103:~/devstack$ cinder list
  +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
  |
   ID
   | Status | Name | Size | Volume Type | Bootable |
   Attached to
   |
  +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
  | 56a63288-5cc0-4f5c-9197-cde731172dd8 | in-use | None | 1 |
   None
   | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e |
  +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
  openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l
  Fri Sep 19 21:38:13 JST 2014
  360060160cf0036002d1475f6e73fe411 dm-2 DGC,VRAID
  size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
  |-+- policy='round-robin 0' prio=-1 status=active
  | |- 4:0:0:42 sdb 8:16 active undef running
  | |- 5:0:0:42 sdd 8:48 active undef running
  | |- 6:0:0:42 sdf 8:80 active undef running
  | `- 7:0:0:42 sdh 8:112 active undef running
  `-+- policy='round-robin 0' prio=-1 status=enabled
  |- 11:0:0:42 sdp 8:240 active undef running
  |- 8:0:0:42 sdj 8:144 active undef running
  |- 9:0:0:42 sdl 8:176 active undef running
  `- 10:0:0:42 sdn 8:208 active undef running
  openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
  Fri Sep 19 21:38:19 JST 2014
  tcp: [10] 172.23.58.228:3260,4 iqn.1992-04.com.emc:cx.fcn00133400150.a7
  tcp: [3] 172.23.58.238:3260,8 iqn.1992-04.com.emc:cx.fcn00133400150.b7
  tcp: [4] 172.23.58.235:3260,20 iqn.1992-04.com.emc:cx.fcn00133400150.b4
  tcp: [5] 172.23.58.236:3260,6 iqn.1992-04.com.emc:cx.fcn00133400150.b5
  tcp: [6] 172.23.58.237:3260,19 iqn.1992-04.com.emc:cx.fcn00133400150.b6
  tcp: [7] 172.23.58.225:3260,16 iqn.1992-04.com.emc:cx.fcn00133400150.a4
  tcp: [8] 172.23.58.226:3260,2 iqn.1992-04.com.emc:cx.fcn00133400150.a5
  tcp: [9] 172.23.58.227:3260,17 iqn.1992-04.com.emc:cx.fcn00133400150.a6

  openstack@W1DEV103:~/devstack$ nova volume-detach 5bd68785-4acf-43ab-ae13-11b1edc3a62e
  56a63288-5cc0-4f5c-9197-cde731172dd8
  openstack@W1DEV103:~/devstack$
  openstack@W1DEV103:~/devstack$ cinder list
  +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
  |
   ID
   | Status | Name | Size | Volume Type | Bootable |
   Attached to
   |
  +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
  | 56a63288-5cc0-4f5c-9197-cde731172dd8 | detaching | None | 1 |
   None
   | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e|

  +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
  openstack@W1DEV103:~/devstack$
  openstack@W1DEV103:~/devstack$ cinder list
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  |
   ID
   | Status | Name | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  | 56a63288-5cc0-4f5c-9197-cde731172dd8 | available | None | 1 |
   None
   | false |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  |
  openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l
  Fri Sep 19 21:39:23 JST 2014
  360060160cf0036002d1475f6e73fe411 dm-2 ,
  size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
  |-+- policy='round-robin 0' prio=-1 status=active
  | |- #:#:#:# - #:# active undef running
  | |- #:#:#:# - #:# active undef running
  | |- #:#:#:# - #:# active undef running
  | `- #:#:#:# - #:# active undef running
  `-+- policy='round-robin 0' prio=-1 status=enabled
  |- #:#:#:# - #:# active undef running
  |- #:#:#:# - #:# active undef running
  |- #:#:#:# - #:# active undef running
  `- #:#:#:# - #:# active undef running
  openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
  Fri Sep 19 21:39:27 JST 2014
  iscsiadm: No active sessions.

  Then I manually removed the multipaths,
  openstack@W1CN103:/etc/iscsi$ sudo multipath -f 360060160cf0036002d1475f6e73fe411
  openstack@W1CN103:/etc/iscsi$ sudo multipath -l
  openstack@W1CN103:/etc/iscsi$

   I think the problem is in,
  virt/libvirt/volume.py:LibvirtISCSIVolumeDriver
   def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device):

  End of this method executes following code to call remove_multipath_device_descriptor for remove the multipaths by multipath -f before return.
          self._remove_multipath_device_descriptor(multipath_device)
          return
  Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one),  this method return without calling _remove_multipath_device_descriptor due to following if statement. 
   
          if not in_use:
              # disconnect if no other multipath devices with same iqn
              self._disconnect_mpath(iscsi_properties, ips_iqns)
              return
  It just disconnect them but not remove them.
  One of the reasons why we have to remove them is,
  https://bugs.launchpad.net/nova/+bug/1223975

  IMO, we should call _remove_multipath_device_descriptor in above if
  statement before return.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1374999/+subscriptions