← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #32861

[Bug 1454621] [NEW] Multipath device descripter is not deleted while the device pathes are removed after detach last volume form VM on a host

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Multipath descripter is not removed in volume detachment

In the iSCSI multipath environment, after I detach a last volume from a
instance on the host. Althougth the devices under /dev/disk/by-path are
removed by the multipath descriptor is not removed. I am using VNX as
cinder backend.

========How I produced this issue=============
1. Before I do an detachment, there is only LUN 86 attached to the host. 
stack@ubuntu-server7:/dev/disk/by-path$ ls
ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-0   pci-0000:04:00.0-scsi-0:1:1:0-part1            pci-0000:05:00.1-fc-0x500601610860080f-lun-0
ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-86  pci-0000:04:00.0-scsi-0:1:1:0-part2            pci-0000:05:00.1-fc-0x50060169086003ba-lun-0
ip-192.168.4.53:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.b6-lun-0   pci-0000:04:00.0-scsi-0:1:1:0-part5            pci-0000:05:00.1-fc-0x50060169086003ba-lun-10
pci-0000:04:00.0-scsi-0:1:0:0                                               pci-0000:05:00.1-fc-0x50060161086003ba-lun-0
pci-0000:04:00.0-scsi-0:1:1:0                                               pci-0000:05:00.1-fc-0x50060161086003ba-lun-10
stack@ubuntu-server7:/dev/disk/by-path$ ls -l
total 0
lrwxrwxrwx 1 root root  9 May 12 23:26 ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-0 -> ../../sdh
lrwxrwxrwx 1 root root  9 May 12 23:26 ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-86 -> ../../sdi
lrwxrwxrwx 1 root root  9 May 12 23:28 ip-192.168.4.53:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.b6-lun-0 -> ../../sdj
lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:04:00.0-scsi-0:1:0:0 -> ../../sdb
lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0 -> ../../sda
lrwxrwxrwx 1 root root 10 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0-part5 -> ../../sda5
lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060161086003ba-lun-0 -> ../../sdc
lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060161086003ba-lun-10 -> ../../sdd
lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x500601610860080f-lun-0 -> ../../sde
lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060169086003ba-lun-0 -> ../../sdf
lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060169086003ba-lun-10 -> ../../sdg
stack@ubuntu-server7:/dev/disk/by-path$ sudo multipath -l /dev/sdi
3600601601bd03200d8f0c1714bf9e411 dm-6 DGC,VRAID
size=3.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=-1 status=active
  `- 12:0:0:86 sdi 8:128 active undef running
  
2. Trigger the detachment
stack@ubuntu-server7:/dev/disk/by-path$ nova volume-detach 818eb5fb-66fe-4179-9466-04ef1ec09a8d 837bd1e1-8d82-472e-bd42-f8edd4fbcc42

3. After the detachment, The devices under /dev/disk/by-path went away this is as excpeted
stack@ubuntu-server7:/dev/disk/by-path$ ls
ip-192.168.4.53:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.b6-lun-0  pci-0000:04:00.0-scsi-0:1:1:0-part2            pci-0000:05:00.1-fc-0x500601610860080f-lun-0
pci-0000:04:00.0-scsi-0:1:0:0                                              pci-0000:04:00.0-scsi-0:1:1:0-part5            pci-0000:05:00.1-fc-0x50060169086003ba-lun-0
pci-0000:04:00.0-scsi-0:1:1:0                                              pci-0000:05:00.1-fc-0x50060161086003ba-lun-0   pci-0000:05:00.1-fc-0x50060169086003ba-lun-10
pci-0000:04:00.0-scsi-0:1:1:0-part1                                        pci-0000:05:00.1-fc-0x50060161086003ba-lun-10

4. But the multipath descripter is not removed 3600601601bd03200d8f0c1714bf9e411
stack@ubuntu-server7:/dev/disk/by-path$ sudo multipath -ll
3600601602ba034002c160db00960e411 dm-1 DGC,VRAID
size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| `- 9:0:2:10 sdg 8:96  active ready  running
`-+- policy='round-robin 0' prio=10 status=enabled
  `- 9:0:0:10 sdd 8:48  active ready  running
3600508e0000000001952e401650d0b0a dm-2 LSI,Logical Volume
size=929G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 6:1:1:0  sda 8:0   active ready  running
3600601601bd03200d8f0c1714bf9e411 dm-6 ,
size=3.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=0 status=active
  `- #:#:#:#  -   #:#   active faulty running
3600508e0000000009551404ce9efae0d dm-0 LSI,Logical Volume
size=222G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  `- 6:1:0:0  sdb 8:16  active ready  running

stack@ubuntu-server7:/dev/disk/by-path$ sudo multipath -l 3600601601bd03200d8f0c1714bf9e411
3600601601bd03200d8f0c1714bf9e411 dm-6 ,
size=3.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=-1 status=active
  `- #:#:#:#  -   #:#   active undef running

======Nova version ==========
stack@ubuntu-server7:/opt/stack/nova$ git log -1
commit 39bbc0d84624d95f13e3d911f38b7ae483c4505e
Author: OpenStack Proposal Bot <openstack-infra@xxxxxxxxxxxxxxxxxxx>
Date:   Sat May 2 06:19:07 2015 +0000

    Imported Translations from Transifex

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: Ifeccc8216e19a2234cd0be983f289fb7a2a0573f


========Problem ========================
Seems the multipath descripter will not be removed automatically after we logout the iscsi targets. multipath -f <multipath id> may need to be invoked explictly.

nova/nova/virt/libvirt/volume.py
in method _disconnect_volume_multipath_iscsi, when there is no other devices attached or the iqn is not used by other devices, it just logout the targets without delete the multipath descriptor

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1454621

Title:
  Multipath device descripter is not deleted while the device pathes are
  removed after detach last volume form VM on a host

Status in OpenStack Compute (Nova):
  New

Bug description:
  Multipath descripter is not removed in volume detachment

  In the iSCSI multipath environment, after I detach a last volume from
  a instance on the host. Althougth the devices under /dev/disk/by-path
  are removed by the multipath descriptor is not removed. I am using VNX
  as cinder backend.

  ========How I produced this issue=============
  1. Before I do an detachment, there is only LUN 86 attached to the host. 
  stack@ubuntu-server7:/dev/disk/by-path$ ls
  ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-0   pci-0000:04:00.0-scsi-0:1:1:0-part1            pci-0000:05:00.1-fc-0x500601610860080f-lun-0
  ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-86  pci-0000:04:00.0-scsi-0:1:1:0-part2            pci-0000:05:00.1-fc-0x50060169086003ba-lun-0
  ip-192.168.4.53:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.b6-lun-0   pci-0000:04:00.0-scsi-0:1:1:0-part5            pci-0000:05:00.1-fc-0x50060169086003ba-lun-10
  pci-0000:04:00.0-scsi-0:1:0:0                                               pci-0000:05:00.1-fc-0x50060161086003ba-lun-0
  pci-0000:04:00.0-scsi-0:1:1:0                                               pci-0000:05:00.1-fc-0x50060161086003ba-lun-10
  stack@ubuntu-server7:/dev/disk/by-path$ ls -l
  total 0
  lrwxrwxrwx 1 root root  9 May 12 23:26 ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-0 -> ../../sdh
  lrwxrwxrwx 1 root root  9 May 12 23:26 ip-192.168.4.52:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.a6-lun-86 -> ../../sdi
  lrwxrwxrwx 1 root root  9 May 12 23:28 ip-192.168.4.53:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.b6-lun-0 -> ../../sdj
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:04:00.0-scsi-0:1:0:0 -> ../../sdb
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0 -> ../../sda
  lrwxrwxrwx 1 root root 10 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0-part1 -> ../../sda1
  lrwxrwxrwx 1 root root 10 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0-part2 -> ../../sda2
  lrwxrwxrwx 1 root root 10 May 12 18:07 pci-0000:04:00.0-scsi-0:1:1:0-part5 -> ../../sda5
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060161086003ba-lun-0 -> ../../sdc
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060161086003ba-lun-10 -> ../../sdd
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x500601610860080f-lun-0 -> ../../sde
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060169086003ba-lun-0 -> ../../sdf
  lrwxrwxrwx 1 root root  9 May 12 18:07 pci-0000:05:00.1-fc-0x50060169086003ba-lun-10 -> ../../sdg
  stack@ubuntu-server7:/dev/disk/by-path$ sudo multipath -l /dev/sdi
  3600601601bd03200d8f0c1714bf9e411 dm-6 DGC,VRAID
  size=3.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  `-+- policy='round-robin 0' prio=-1 status=active
    `- 12:0:0:86 sdi 8:128 active undef running
    
  2. Trigger the detachment
  stack@ubuntu-server7:/dev/disk/by-path$ nova volume-detach 818eb5fb-66fe-4179-9466-04ef1ec09a8d 837bd1e1-8d82-472e-bd42-f8edd4fbcc42

  3. After the detachment, The devices under /dev/disk/by-path went away this is as excpeted
  stack@ubuntu-server7:/dev/disk/by-path$ ls
  ip-192.168.4.53:3260-iscsi-iqn.1992-04.com.emc:cx.fnm00130200235.b6-lun-0  pci-0000:04:00.0-scsi-0:1:1:0-part2            pci-0000:05:00.1-fc-0x500601610860080f-lun-0
  pci-0000:04:00.0-scsi-0:1:0:0                                              pci-0000:04:00.0-scsi-0:1:1:0-part5            pci-0000:05:00.1-fc-0x50060169086003ba-lun-0
  pci-0000:04:00.0-scsi-0:1:1:0                                              pci-0000:05:00.1-fc-0x50060161086003ba-lun-0   pci-0000:05:00.1-fc-0x50060169086003ba-lun-10
  pci-0000:04:00.0-scsi-0:1:1:0-part1                                        pci-0000:05:00.1-fc-0x50060161086003ba-lun-10

  4. But the multipath descripter is not removed 3600601601bd03200d8f0c1714bf9e411
  stack@ubuntu-server7:/dev/disk/by-path$ sudo multipath -ll
  3600601602ba034002c160db00960e411 dm-1 DGC,VRAID
  size=1.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  |-+- policy='round-robin 0' prio=130 status=active
  | `- 9:0:2:10 sdg 8:96  active ready  running
  `-+- policy='round-robin 0' prio=10 status=enabled
    `- 9:0:0:10 sdd 8:48  active ready  running
  3600508e0000000001952e401650d0b0a dm-2 LSI,Logical Volume
  size=929G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=1 status=active
    `- 6:1:1:0  sda 8:0   active ready  running
  3600601601bd03200d8f0c1714bf9e411 dm-6 ,
  size=3.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  `-+- policy='round-robin 0' prio=0 status=active
    `- #:#:#:#  -   #:#   active faulty running
  3600508e0000000009551404ce9efae0d dm-0 LSI,Logical Volume
  size=222G features='0' hwhandler='0' wp=rw
  `-+- policy='round-robin 0' prio=1 status=active
    `- 6:1:0:0  sdb 8:16  active ready  running

  stack@ubuntu-server7:/dev/disk/by-path$ sudo multipath -l 3600601601bd03200d8f0c1714bf9e411
  3600601601bd03200d8f0c1714bf9e411 dm-6 ,
  size=3.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  `-+- policy='round-robin 0' prio=-1 status=active
    `- #:#:#:#  -   #:#   active undef running

  ======Nova version ==========
  stack@ubuntu-server7:/opt/stack/nova$ git log -1
  commit 39bbc0d84624d95f13e3d911f38b7ae483c4505e
  Author: OpenStack Proposal Bot <openstack-infra@xxxxxxxxxxxxxxxxxxx>
  Date:   Sat May 2 06:19:07 2015 +0000

      Imported Translations from Transifex

      For more information about this automatic import see:
      https://wiki.openstack.org/wiki/Translations/Infrastructure

      Change-Id: Ifeccc8216e19a2234cd0be983f289fb7a2a0573f

  
  ========Problem ========================
  Seems the multipath descripter will not be removed automatically after we logout the iscsi targets. multipath -f <multipath id> may need to be invoked explictly.

  nova/nova/virt/libvirt/volume.py
  in method _disconnect_volume_multipath_iscsi, when there is no other devices attached or the iqn is not used by other devices, it just logout the targets without delete the multipath descriptor

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1454621/+subscriptions

Follow ups

References

  Thread Previous • Date Previous • • Thread Next