yahoo-eng-team team mailing list archive

[Launchpad Bug Tracker <1374999@xxxxxxxxxxxxxxxxxx>]

This bug was fixed in the package nova - 1:2014.1.5-0ubuntu1.5

---------------
nova (1:2014.1.5-0ubuntu1.5) trusty; urgency=medium

  * Fix live migration usage of the wrong connector (LP: #1475411)
    - d/p/Fix-live-migrations-usage-of-the-wrong-connector-inf.patch
  * Fix wrong used ProcessExecutionError exception (LP: #1308839)
    - d/p/Fix-wrong-used-ProcessExecutionError-exception.patch
  * Clean up iSCSI multipath devices in Post Live Migration (LP: #1357368)
    - d/p/Clean-up-iSCSI-multipath-devices-in-Post-Live-Migrat.patch
  * Detach iSCSI latest path for latest disk (LP: #1374999)
    - d/p/Detach-iSCSI-latest-path-for-latest-disk.patch

 -- Billy Olsen <billy.olsen@xxxxxxxxxxxxx>  Fri, 29 Apr 2016 15:35:01
-0700

** Changed in: nova (Ubuntu Trusty)
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1374999

Title:
  iSCSI volume detach does not correctly remove the multipath device
  descriptors

Status in Ubuntu Cloud Archive:
  Confirmed
Status in Ubuntu Cloud Archive icehouse series:
  Fix Committed
Status in Ubuntu Cloud Archive juno series:
  Won't Fix
Status in Ubuntu Cloud Archive kilo series:
  Fix Released
Status in OpenStack Compute (nova):
  Invalid
Status in nova package in Ubuntu:
  Fix Released
Status in nova source package in Trusty:
  Fix Released

Bug description:
  [Impact]

  iSCSI volume detach does not correctly remove the multipath device
  descriptors.

  The multipath devices are left on the compute node and multipath tools
  will occaisionally send IOs to known multipath devices.

  [Test Case]

  tested environment:
  nova-compute on Ubuntu 14.04.1, iscsi_use_multipath=True and iSCSI volume backend is EMC VNX 5300.

   I created 3 cinder volumes and attached them to a nova instance. Then I detach them one by one. First 2 volumes volumes detached successfully.  3rd volume also successfully detached but ended up with  failed multipaths.
  Here is the terminal log for last volume detach.

  openstack@W1DEV103:~/devstack$ cinder list
  +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
  |
   ID
   | Status | Name | Size | Volume Type | Bootable |
   Attached to
   |
  +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
  | 56a63288-5cc0-4f5c-9197-cde731172dd8 | in-use | None | 1 |
   None
   | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e |
  +--------------------------------------+--------+------+------+-------------+----------+--------------------------------------+
  openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l
  Fri Sep 19 21:38:13 JST 2014
  360060160cf0036002d1475f6e73fe411 dm-2 DGC,VRAID
  size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
  |-+- policy='round-robin 0' prio=-1 status=active
  | |- 4:0:0:42 sdb 8:16 active undef running
  | |- 5:0:0:42 sdd 8:48 active undef running
  | |- 6:0:0:42 sdf 8:80 active undef running
  | `- 7:0:0:42 sdh 8:112 active undef running
  `-+- policy='round-robin 0' prio=-1 status=enabled
  |- 11:0:0:42 sdp 8:240 active undef running
  |- 8:0:0:42 sdj 8:144 active undef running
  |- 9:0:0:42 sdl 8:176 active undef running
  `- 10:0:0:42 sdn 8:208 active undef running
  openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
  Fri Sep 19 21:38:19 JST 2014
  tcp: [10] 172.23.58.228:3260,4 iqn.1992-04.com.emc:cx.fcn00133400150.a7
  tcp: [3] 172.23.58.238:3260,8 iqn.1992-04.com.emc:cx.fcn00133400150.b7
  tcp: [4] 172.23.58.235:3260,20 iqn.1992-04.com.emc:cx.fcn00133400150.b4
  tcp: [5] 172.23.58.236:3260,6 iqn.1992-04.com.emc:cx.fcn00133400150.b5
  tcp: [6] 172.23.58.237:3260,19 iqn.1992-04.com.emc:cx.fcn00133400150.b6
  tcp: [7] 172.23.58.225:3260,16 iqn.1992-04.com.emc:cx.fcn00133400150.a4
  tcp: [8] 172.23.58.226:3260,2 iqn.1992-04.com.emc:cx.fcn00133400150.a5
  tcp: [9] 172.23.58.227:3260,17 iqn.1992-04.com.emc:cx.fcn00133400150.a6

  openstack@W1DEV103:~/devstack$ nova volume-detach 5bd68785-4acf-43ab-ae13-11b1edc3a62e
  56a63288-5cc0-4f5c-9197-cde731172dd8
  openstack@W1DEV103:~/devstack$
  openstack@W1DEV103:~/devstack$ cinder list
  +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
  |
   ID
   | Status | Name | Size | Volume Type | Bootable |
   Attached to
   |
  +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
  | 56a63288-5cc0-4f5c-9197-cde731172dd8 | detaching | None | 1 |
   None
   | false | 5bd68785-4acf-43ab-ae13-11b1edc3a62e|

  +--------------------------------------+-----------+------+------+-------------+----------+--------------------------------------+
  openstack@W1DEV103:~/devstack$
  openstack@W1DEV103:~/devstack$ cinder list
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  |
   ID
   | Status | Name | Size | Volume Type | Bootable | Attached to |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  | 56a63288-5cc0-4f5c-9197-cde731172dd8 | available | None | 1 |
   None
   | false |
  +--------------------------------------+-----------+------+------+-------------+----------+-------------+
  |
  openstack@W1CN103:/etc/iscsi$ date;sudo multipath -l
  Fri Sep 19 21:39:23 JST 2014
  360060160cf0036002d1475f6e73fe411 dm-2 ,
  size=1.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
  |-+- policy='round-robin 0' prio=-1 status=active
  | |- #:#:#:# - #:# active undef running
  | |- #:#:#:# - #:# active undef running
  | |- #:#:#:# - #:# active undef running
  | `- #:#:#:# - #:# active undef running
  `-+- policy='round-robin 0' prio=-1 status=enabled
  |- #:#:#:# - #:# active undef running
  |- #:#:#:# - #:# active undef running
  |- #:#:#:# - #:# active undef running
  `- #:#:#:# - #:# active undef running
  openstack@W1CN103:/etc/iscsi$ date;sudo iscsiadm -m session
  Fri Sep 19 21:39:27 JST 2014
  iscsiadm: No active sessions.

  Then I manually removed the multipaths,
  openstack@W1CN103:/etc/iscsi$ sudo multipath -f 360060160cf0036002d1475f6e73fe411
  openstack@W1CN103:/etc/iscsi$ sudo multipath -l
  openstack@W1CN103:/etc/iscsi$

   I think the problem is in,
  virt/libvirt/volume.py:LibvirtISCSIVolumeDriver
   def _disconnect_volume_multipath_iscsi(self, iscsi_properties, multipath_device):

  End of this method executes following code to call remove_multipath_device_descriptor for remove the multipaths by multipath -f before return.
          self._remove_multipath_device_descriptor(multipath_device)
          return
  Therefore, first two volumes worked fine. However, when it comes to the last device (in this case 3rd one),  this method return without calling _remove_multipath_device_descriptor due to following if statement.

          if not in_use:
              # disconnect if no other multipath devices with same iqn
              self._disconnect_mpath(iscsi_properties, ips_iqns)
              return
  It just disconnect them but not remove them.
  One of the reasons why we have to remove them is,
  https://bugs.launchpad.net/nova/+bug/1223975

  IMO, we should call _remove_multipath_device_descriptor in above if
  statement before return.

  [Regression Potential]

  - Low: the change is relatively minor and allows the code to also
  remove the device when the last path is removed. Should a regression
  occur, it should be limited to iscsi-multipath device detachment,
  which is a small portion of installations.

To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1374999/+subscriptions