yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1454621] Re: Multipath device descripter is not deleted while the device pathes are removed after detach last volume form VM on a host

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Wed, 23 Sep 2015 18:40:32 -0000
Reply-to: Bug 1454621 <1454621@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Is this still an issue in liberty?

** Also affects: os-brick
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1454621

Title:
  Multipath device descripter is not deleted while the device pathes are
  removed after detach last volume form VM on a host

Status in OpenStack Compute (nova):
  New
Status in os-brick:
  New

Bug description:
  Multipath descripter is not removed in volume detachment

  In the iSCSI multipath environment, after I detach a last volume from
  a instance on the host. Althougth the devices under /dev/disk/by-path
  are removed by the multipath descriptor is not removed. I am using VNX
  as cinder backend.

  ========How I produced this issue=============
  1. Before I do an detachment, there is only LUN 86 attached to the host. 
  stack@ubuntu-server7:/dev/disk/by-path$ ls
  ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-0   pci-0000:04:00.0-scsi-0:1:1:0-part1            pci-0000:05:00.1-fc-0x500601610860080f-lun-0
  ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-86  pci-0000:04:00.0-scsi-0:1:1:0-part2            pci-0000:05:00.1-fc-0x50060169086003ba-lun-0
  ip-192.168.4.53:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.b6-lun-0   pci-0000:04:00.0-scsi-0:1:1:0-part5            pci-0000:05:00.1-fc-0x50060169086003ba-lun-10
  pci-0000:04:00.0-scsi-0:1:0:0                                               pci-0000:05:00.1-fc-0x50060161086003ba-lun-0
  pci-0000:04:00.0-scsi-0:1:1:0                                               pci-0000:05:00.1-fc-0x50060161086003ba-lun-10
  stack@ubuntu-server7:/dev/disk/by-path$ ls -l
  total 0
  lrwxrwxrwx 1 root root  9 May 12 23:26 ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-0 -> ../../sdh
  lrwxrwxrwx 1 root root  9 May 12 23:26 ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-86 -> ../../sdi
  lrwxrwxrwx 1 root root  9 May 12 23:28 ip-192.168.4.53:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.b6-lun-0 -> ../../sdj
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:04:00.0-scsi-0:1:0:0 -> ../../sdb
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0 -> ../../sda
  lrwxrwxrwx 1 root root 10 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0-part1 -> ../../sda1
  lrwxrwxrwx 1 root root 10 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0-part2 -> ../../sda2
  lrwxrwxrwx 1 root root 10 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0-part5 -> ../../sda5
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060161086003ba-lun-0 -> ../../sdc
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060161086003ba-lun-10 -> ../../sdd
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x500601610860080f-lun-0 -> ../../sde
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060169086003ba-lun-0 -> ../../sdf
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060169086003ba-lun-10 -> ../../sdg
  stack@ubuntu-server7:/dev/disk/by-path$ sudo multipath -l /dev/sdi
  3600601601bd03200d8f0c1714bf9e411 dm-6 DGC,VRAID
  size=3.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  `-+- policy='round-robin 0' prio=-1 status=active
    `- 12:0:0:86 sdi 8:128 active undef running
    
  2. Trigger the detachment
  stack@ubuntu-server7:/dev/disk/by-path$ nova volume-detach 818eb5fb-66fe-4179-9466-04ef1ec09a8d 837bd1e1-8d82-472e-bd42-f8edd4fbcc42

  3. After the detachment, The devices under /dev/disk/by-path went away this is as excpeted
  stack@ubuntu-server7:/dev/disk/by-path$ ls
  ip-192.168.4.53:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.b6-lun-0  pci-0000:04:00.0-scsi-0:1:1:0-part2            pci-0000:05:00.1-fc-0x500601610860080f-lun-0
  pci-0000:04:00.0-scsi-0:1:0:0                                              pci-0000:04:00.0-scsi-0:1:1:0-part5            pci-0000:05:00.1-fc-0x50060169086003ba-lun-0
  pci-0000:04:00.0-scsi-0:1:1:0                                              pci-0000:05:00.1-fc-0x50060161086003ba-lun-0   pci-0000:05:00.1-fc-0x50060169086003ba-lun-10
  pci-0000:04:00.0-scsi-0:1:1:0-part1                                        pci-0000:05:00.1-fc-0x50060161086003ba-lun-10

  4. But the multipath descripter is not removed 3600601601bd03200d8f0c1714bf9e411
  stack@ubuntu-server7:/dev/disk/by-path$ sudo multipath -ll
  3600601602ba034002c160db00960e411 dm-1 DGC,VRAID
  size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  |-+- policy='round-robin 0' prio=130 status=active
  | `- 9:0:2:10 sdg 8:96  active ready  running
  `-+- policy='round-robin 0' prio=10 status=enabled
    `- 9:0:0:10 sdd 8:48  active ready  running
  3600508e0000000001952e401650d0b0a dm-2 LSI,Logical Volume
  size=929G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=1 status=active
    `- 6:1:1:0  sda 8:0   active ready  running
  3600601601bd03200d8f0c1714bf9e411 dm-6 ,
  size=3.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  `-+- policy='round-robin 0' prio=0 status=active
    `- #:#:#:#  -   #:#   active faulty running
  3600508e0000000009551404ce9efae0d dm-0 LSI,Logical Volume
  size=222G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=1 status=active
    `- 6:1:0:0  sdb 8:16  active ready  running

  stack@ubuntu-server7:/dev/disk/by-path$ sudo multipath -l 3600601601bd03200d8f0c1714bf9e411
  3600601601bd03200d8f0c1714bf9e411 dm-6 ,
  size=3.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  `-+- policy='round-robin 0' prio=-1 status=active
    `- #:#:#:#  -   #:#   active undef running

  ======Nova version ==========
  stack@ubuntu-server7:/opt/stack/nova$ git log -1
  commit 39bbc0d84624d95f13e3d911f38b7ae483c4505e
  Author: OpenStack Proposal Bot <openstack-infra@xxxxxxxxxxxxxxxxxxx>
  Date:   Sat May 2 06:19:07 2015 +0000

      Imported Translations from Transifex

      For more information about this automatic import see:
      https://wiki.openstack.org/wiki/Translations/Infrastructure

      Change-Id: Ifeccc8216e19a2234cd0be983f289fb7a2a0573f

  
  ========Problem ========================
  Seems the multipath descripter will not be removed automatically after we logout the iscsi targets. multipath -f <multipath id> may need to be invoked explictly.

  nova/nova/virt/libvirt/volume.py
  in method _disconnect_volume_multipath_iscsi, when there is no other devices attached or the iqn is not used by other devices, it just logout the targets without delete the multipath descriptor

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1454621/+subscriptions
References

[Bug 1454621] [NEW] Multipath device descripter is not deleted while the device pathes are removed after detach last volume form VM on a host
From: Tina Tang, 2015-05-13