yahoo-eng-team team mailing list archive

[Tina Tang <tina.tang@xxxxxxx>]

Public bug reported:

We detacted this during our test. Sometimes, there maybe error string
comes out in the begining of the output of multipath -ll <device>. See
below log:

2015-05-11 05:59:53.005 DEBUG oslo_concurrency.processutils [req-30af6607-6e51-487e-beb2-3dbea8ee9fac admin admin] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll /dev/sdfb" returned: 0 in 0.912s execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:225
2015-05-11 05:59:53.005 DEBUG nova.virt.libvirt.volume [req-30af6607-6e51-487e-beb2-3dbea8ee9fac admin admin] multipath ['-ll', u'/dev/sdfb']: stdout=May 11 05:59:52 | io_setup failed
3600601602ba0340035339627c3f7e411 dm-44 DGC,VRAID
size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
|-+- policy='round-robin 0' prio=130 status=active
| `- 34:0:0:193 sdfb 129:208 active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 35:0:0:193 sdfc 129:224 active ready running
  `- 36:0:0:193 sdfd 129:240 active ready running

When this error appears, the multipath id of a device will be
incorrectly parsed. In above instance, the multipath device name was set
to "/dev/mapper/May" which is incorrect.

 nova/virt/libvirt/volume.py
 744 def _get_multipath_device_name(self, single_path_device):
 745 device = os.path.realpath(single_path_device)
 746
 747 out = self._run_multipath(['-ll',
 748 device],
 749 check_exit_code=[0, 1])[0]
 750 mpath_line = [line for line in out.splitlines()
 751 if "scsi_id" not in line] # ignore udev errors
 752 if len(mpath_line) > 0 and len(mpath_line[0]) > 0:
 753 return "/dev/mapper/%s" % mpath_line[0].split(" ")[0]
 754
 755 return None

stack@openstack-performance:~/tina/nova_iscsi_mp/nova$ git log -1
commit f4504f3575b35ec14390b4b678e441fcf953f47b
Merge: 3f21f60 5fbd852
Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
Date: Tue May 12 22:46:43 2015 +0000

    Merge "Remove db layer hard-code permission checks for
network_get_all_by_host"

** Affects: nova
     Importance: Undecided
         Status: New

** Description changed:

  We detacted this during our test. Sometimes, there maybe error string
- comes out in the begining of the output of multipath -ll <device>.  See
+ comes out in the begining of the output of multipath -ll <device>. See
  below log:
  
  2015-05-11 05:59:53.005 DEBUG oslo_concurrency.processutils [req-30af6607-6e51-487e-beb2-3dbea8ee9fac admin admin] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll /dev/sdfb" returned: 0 in 0.912s execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:225
  2015-05-11 05:59:53.005 DEBUG nova.virt.libvirt.volume [req-30af6607-6e51-487e-beb2-3dbea8ee9fac admin admin] multipath ['-ll', u'/dev/sdfb']: stdout=May 11 05:59:52 | io_setup failed
  3600601602ba0340035339627c3f7e411 dm-44 DGC,VRAID
  size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  |-+- policy='round-robin 0' prio=130 status=active
- | `- 34:0:0:193 sdfb 129:208 active ready  running
+ | `- 34:0:0:193 sdfb 129:208 active ready running
  `-+- policy='round-robin 0' prio=10 status=enabled
-   |- 35:0:0:193 sdfc 129:224 active ready  running
-   `- 36:0:0:193 sdfd 129:240 active ready  running
-  
- When this error appears, the multipath id of a device will be incorrectly parsed. In above instance, the multipath device name was set to "/dev/mapper/May" which is incorrect.
+   |- 35:0:0:193 sdfc 129:224 active ready running
+   `- 36:0:0:193 sdfd 129:240 active ready running
  
+ When this error appears, the multipath id of a device will be
+ incorrectly parsed. In above instance, the multipath device name was set
+ to "/dev/mapper/May" which is incorrect.
  
-  744     def _get_multipath_device_name(self, single_path_device):
-  745         device = os.path.realpath(single_path_device)
+  nova/virt/libvirt/volume.py
+  744 def _get_multipath_device_name(self, single_path_device):
+  745 device = os.path.realpath(single_path_device)
   746
-  747         out = self._run_multipath(['-ll',
-  748                                   device],
-  749                                   check_exit_code=[0, 1])[0]
-  750         mpath_line = [line for line in out.splitlines()
-  751                       if "scsi_id" not in line]  # ignore udev errors
-  752         if len(mpath_line) > 0 and len(mpath_line[0]) > 0:
-  753             return "/dev/mapper/%s" % mpath_line[0].split(" ")[0]
+  747 out = self._run_multipath(['-ll',
+  748 device],
+  749 check_exit_code=[0, 1])[0]
+  750 mpath_line = [line for line in out.splitlines()
+  751 if "scsi_id" not in line] # ignore udev errors
+  752 if len(mpath_line) > 0 and len(mpath_line[0]) > 0:
+  753 return "/dev/mapper/%s" % mpath_line[0].split(" ")[0]
   754
-  755         return None
- 
+  755 return None
  
  stack@openstack-performance:~/tina/nova_iscsi_mp/nova$ git log -1
  commit f4504f3575b35ec14390b4b678e441fcf953f47b
  Merge: 3f21f60 5fbd852
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
- Date:   Tue May 12 22:46:43 2015 +0000
+ Date: Tue May 12 22:46:43 2015 +0000
  
      Merge "Remove db layer hard-code permission checks for
  network_get_all_by_host"

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1454519

Title:
  iSCSI Multipath, multipath id is incorrect parsed when "io_setup
  failed" error appears in multipath -ll output

Status in OpenStack Compute (Nova):
  New

Bug description:
  We detacted this during our test. Sometimes, there maybe error string
  comes out in the begining of the output of multipath -ll <device>. See
  below log:

  2015-05-11 05:59:53.005 DEBUG oslo_concurrency.processutils [req-30af6607-6e51-487e-beb2-3dbea8ee9fac admin admin] CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -ll /dev/sdfb" returned: 0 in 0.912s execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:225
  2015-05-11 05:59:53.005 DEBUG nova.virt.libvirt.volume [req-30af6607-6e51-487e-beb2-3dbea8ee9fac admin admin] multipath ['-ll', u'/dev/sdfb']: stdout=May 11 05:59:52 | io_setup failed
  3600601602ba0340035339627c3f7e411 dm-44 DGC,VRAID
  size=5.0G features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
  |-+- policy='round-robin 0' prio=130 status=active
  | `- 34:0:0:193 sdfb 129:208 active ready running
  `-+- policy='round-robin 0' prio=10 status=enabled
    |- 35:0:0:193 sdfc 129:224 active ready running
    `- 36:0:0:193 sdfd 129:240 active ready running

  When this error appears, the multipath id of a device will be
  incorrectly parsed. In above instance, the multipath device name was
  set to "/dev/mapper/May" which is incorrect.

   nova/virt/libvirt/volume.py
   744 def _get_multipath_device_name(self, single_path_device):
   745 device = os.path.realpath(single_path_device)
   746
   747 out = self._run_multipath(['-ll',
   748 device],
   749 check_exit_code=[0, 1])[0]
   750 mpath_line = [line for line in out.splitlines()
   751 if "scsi_id" not in line] # ignore udev errors
   752 if len(mpath_line) > 0 and len(mpath_line[0]) > 0:
   753 return "/dev/mapper/%s" % mpath_line[0].split(" ")[0]
   754
   755 return None

  stack@openstack-performance:~/tina/nova_iscsi_mp/nova$ git log -1
  commit f4504f3575b35ec14390b4b678e441fcf953f47b
  Merge: 3f21f60 5fbd852
  Author: Jenkins <jenkins@xxxxxxxxxxxxxxxxxxxx>
  Date: Tue May 12 22:46:43 2015 +0000

      Merge "Remove db layer hard-code permission checks for
  network_get_all_by_host"

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1454519/+subscriptions