yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1370226] [NEW] LibvirtISCSIVolumeDriver cannot find volumes that include pci-* in the /dev/disk/by-path device

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jay Bryant <jsbryant@xxxxxxxxxx>
Date: Tue, 16 Sep 2014 20:05:13 -0000
Reply-to: Bug 1370226 <1370226@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

I am currently unable to attach iSCSI volumes to our system because the
path that is expected by the LibvirtISCSIVolumeDriver doesn't match what
is being created in /dev/disk/by-path:

2014-09-16 01:33:22.533 24304 DEBUG nova.openstack.common.lockutils [req-f466db73-0a7c-4e1f-85ad-473c688d0a68 None] Semaphore / lock released "connect_volume" inner /usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py:328
2014-09-16 01:33:22.534 24304 ERROR nova.virt.block_device [req-f466db73-0a7c-4e1f-85ad-473c688d0a68 None] [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9] Driver failed to attach volume 97e38815-c934-48a7-b343-880c5a9bf4b8 at /dev/vdd
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9] Traceback (most recent call last):
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 252, in attach
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     device_type=self['device_type'], encryption=encryption)
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1283, in attach_volume
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     conf = self._connect_volume(connection_info, disk_info)
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1237, in _connect_volume
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     return driver.connect_volume(connection_info, disk_info)
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 325, in inner
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     return f(*args, **kwargs)
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 295, in connect_volume
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     % (host_device))
2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9] NovaException: iSCSI device not found at /dev/disk/by-path/ip-10.90.50.10:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node2-lun-4

The paths that are being created, however, are  of the following format:

[root@abba-n09 rules.d]# ll /dev/disk/by-path/
total 0
lrwxrwxrwx. 1 root root  9 Sep 16 10:56 pci-0000:0c:00.2-ip-10.90.50.11:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node1-lun-0 -> ../../sdc
lrwxrwxrwx. 1 root root  9 Sep 16 10:56 pci-0000:0c:00.2-ip-10.90.50.11:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node1-lun-1 -> ../../sdd
lrwxrwxrwx. 1 root root  9 Sep 16 10:56 pci-0000:0c:00.2-ip-10.90.50.11:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node1-lun-2 -> ../../sde
lrwxrwxrwx. 1 root root  9 Sep 16 10:56 pci-0000:0c:00.2-ip-10.90.50.11:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node1-lun-3 -> ../../sdf
lrwxrwxrwx. 1 root root  9 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0 -> ../../sda
lrwxrwxrwx. 1 root root 10 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0-part2 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0-part3 -> ../../sda3
lrwxrwxrwx. 1 root root 10 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0-part4 -> ../../sda4
lrwxrwxrwx. 1 root root  9 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:1:0 -> ../../sdb
[root@abba-n09 rules.d]#

When the devices are created the physical location of the HBA is being included:
0c:00.2 Mass storage controller: Emulex Corporation OneConnect 10Gb iSCSI Initiator (be3) (rev 02)

Looking at the code, I see that theLibvirtISERVolumeDriver actually does
the check that accounts for this  /dev/disk/by-path formatting in the
_get_host_device function:

    def _get_host_device(self, iser_properties):
        time.sleep(1)
        host_device = None
        device = ("ip-%s-iscsi-%s-lun-%s" %
                  (iser_properties['target_portal'],
                   iser_properties['target_iqn'],
                   iser_properties.get('target_lun', 0)))
        look_for_device = glob.glob('/dev/disk/by-path/*%s' % device)
        if look_for_device:
            host_device = look_for_device[0]
        return host_device

So, I was able to get the volume to mount properly by changing the nova.conf file with the following change:
volume_drivers=iscsi=nova.virt.libvirt.volume.LibvirtISERVolumeDriver

Setting the iscsi driver, however, to use the iser driver seems
suspicious to me.  It seems like we still have a bug here somewhere.

The node where I am seeing this is using HBA's.  So maybe the iser
driver is the right option.  Does this mean, however that the
storwize_svc driver that is being used to create the volumes should have
an iSER option?

Should the iSCSI driver be updated to do   look_for_device =
glob.glob('/dev/disk/by-path/*%s' % device) or is it a valid config to
do iscsi=nova.virt.libvirt.volume.LibvirtISERVolumeDriver ?

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1370226

Title:
  LibvirtISCSIVolumeDriver cannot find volumes that include pci-* in the
  /dev/disk/by-path device

Status in OpenStack Compute (Nova):
  New

Bug description:
  I am currently unable to attach iSCSI volumes to our system because
  the path that is expected by the LibvirtISCSIVolumeDriver doesn't
  match what is being created in /dev/disk/by-path:

  2014-09-16 01:33:22.533 24304 DEBUG nova.openstack.common.lockutils [req-f466db73-0a7c-4e1f-85ad-473c688d0a68 None] Semaphore / lock released "connect_volume" inner /usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py:328
  2014-09-16 01:33:22.534 24304 ERROR nova.virt.block_device [req-f466db73-0a7c-4e1f-85ad-473c688d0a68 None] [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9] Driver failed to attach volume 97e38815-c934-48a7-b343-880c5a9bf4b8 at /dev/vdd
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9] Traceback (most recent call last):
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/virt/block_device.py", line 252, in attach
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     device_type=self['device_type'], encryption=encryption)
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1283, in attach_volume
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     conf = self._connect_volume(connection_info, disk_info)
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 1237, in _connect_volume
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     return driver.connect_volume(connection_info, disk_info)
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 325, in inner
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     return f(*args, **kwargs)
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/volume.py", line 295, in connect_volume
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9]     % (host_device))
  2014-09-16 01:33:22.534 24304 TRACE nova.virt.block_device [instance: 097e5a6a-ed49-4914-a0ed-5d58959594c9] NovaException: iSCSI device not found at /dev/disk/by-path/ip-10.90.50.10:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node2-lun-4

  The paths that are being created, however, are  of the following
  format:

  [root@abba-n09 rules.d]# ll /dev/disk/by-path/
  total 0
  lrwxrwxrwx. 1 root root  9 Sep 16 10:56 pci-0000:0c:00.2-ip-10.90.50.11:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node1-lun-0 -> ../../sdc
  lrwxrwxrwx. 1 root root  9 Sep 16 10:56 pci-0000:0c:00.2-ip-10.90.50.11:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node1-lun-1 -> ../../sdd
  lrwxrwxrwx. 1 root root  9 Sep 16 10:56 pci-0000:0c:00.2-ip-10.90.50.11:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node1-lun-2 -> ../../sde
  lrwxrwxrwx. 1 root root  9 Sep 16 10:56 pci-0000:0c:00.2-ip-10.90.50.11:3260-iscsi-iqn.1986-03.com.ibm:2145.abbav3700.node1-lun-3 -> ../../sdf
  lrwxrwxrwx. 1 root root  9 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0 -> ../../sda
  lrwxrwxrwx. 1 root root 10 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0-part1 -> ../../sda1
  lrwxrwxrwx. 1 root root 10 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0-part2 -> ../../sda2
  lrwxrwxrwx. 1 root root 10 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0-part3 -> ../../sda3
  lrwxrwxrwx. 1 root root 10 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:0:0-part4 -> ../../sda4
  lrwxrwxrwx. 1 root root  9 Sep 10 18:46 pci-0000:16:00.0-scsi-0:2:1:0 -> ../../sdb
  [root@abba-n09 rules.d]#

  When the devices are created the physical location of the HBA is being included:
  0c:00.2 Mass storage controller: Emulex Corporation OneConnect 10Gb iSCSI Initiator (be3) (rev 02)

  Looking at the code, I see that theLibvirtISERVolumeDriver actually
  does the check that accounts for this  /dev/disk/by-path formatting in
  the _get_host_device function:

      def _get_host_device(self, iser_properties):
          time.sleep(1)
          host_device = None
          device = ("ip-%s-iscsi-%s-lun-%s" %
                    (iser_properties['target_portal'],
                     iser_properties['target_iqn'],
                     iser_properties.get('target_lun', 0)))
          look_for_device = glob.glob('/dev/disk/by-path/*%s' % device)
          if look_for_device:
              host_device = look_for_device[0]
          return host_device

  So, I was able to get the volume to mount properly by changing the nova.conf file with the following change:
  volume_drivers=iscsi=nova.virt.libvirt.volume.LibvirtISERVolumeDriver

  Setting the iscsi driver, however, to use the iser driver seems
  suspicious to me.  It seems like we still have a bug here somewhere.

  The node where I am seeing this is using HBA's.  So maybe the iser
  driver is the right option.  Does this mean, however that the
  storwize_svc driver that is being used to create the volumes should
  have an iSER option?

  Should the iSCSI driver be updated to do   look_for_device =
  glob.glob('/dev/disk/by-path/*%s' % device) or is it a valid config to
  do iscsi=nova.virt.libvirt.volume.LibvirtISERVolumeDriver ?

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1370226/+subscriptions
Follow ups

[Bug 1370226] Re: LibvirtISCSIVolumeDriver cannot find volumes that include pci-* in the /dev/disk/by-path device
From: Doug Hellmann, 2015-09-02
[Bug 1370226] Re: LibvirtISCSIVolumeDriver cannot find volumes that include pci-* in the /dev/disk/by-path device
From: Walt Boring, 2015-04-15
[Bug 1370226] Re: LibvirtISCSIVolumeDriver cannot find volumes that include pci-* in the /dev/disk/by-path device
From: Thierry Carrez, 2015-02-05
[Bug 1370226] [NEW] LibvirtISCSIVolumeDriver cannot find volumes that include pci-* in the /dev/disk/by-path device
From: Jay Bryant, 2014-09-16
References

[Bug 1370226] [NEW] LibvirtISCSIVolumeDriver cannot find volumes that include pci-* in the /dev/disk/by-path device
From: Jay Bryant, 2014-09-16