yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1424968] Re: unhandled return from linuxscsi.find_multipath_device in disconnect_volume

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Thierry Carrez <thierry.carrez+lp@xxxxxxxxx>
Date: Wed, 24 Jun 2015 12:20:10 -0000
Reply-to: Bug 1424968 <1424968@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
** Changed in: nova
       Status: Fix Committed => Fix Released

** Changed in: nova
    Milestone: None => liberty-1

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1424968

Title:
  unhandled return from linuxscsi.find_multipath_device in
  disconnect_volume

Status in OpenStack Compute (Nova):
  Fix Released

Bug description:
  disconnect_volume in class LibvirtFibreChannelVolumeDriver calls
  linuxscsi.find_multipath_device but does not handle the potential
  return of None (eg if the multipath device does not exist)

  https://git.openstack.org/cgit/openstack/nova/tree/nova/virt/libvirt/volume.py?id=1469c8e14267e27ecc6ced29c91dc1506ce26633#n992

  https://git.openstack.org/cgit/openstack/nova/tree/nova/storage/linuxscsi.py?id=1469c8e14267e27ecc6ced29c91dc1506ce26633#n91

  
  Adding the following to disconnect_volume resolved the issue for me:

           if 'multipath_id' in connection_info['data']:
               multipath_id = connection_info['data']['multipath_id']
               mdev_info = linuxscsi.find_multipath_device(multipath_id)
  +            if mdev_info is None:
  +                return
               devices = mdev_info['devices']
               LOG.debug("devices to remove = %s", devices)

  
  Nova Logs from Juno 

  2015-02-24 17:48:01.140 32199 AUDIT nova.service [-] Starting compute node (version 2014.2.1-1.el7.centos)
  2015-02-24 17:48:01.264 32199 INFO nova.compute.manager [-] [instance: 210f6191-a646-4bba-96ef-80df63d2df01] Deleting instance as its host (Compute-02.domain) is not equal to our host (Compute-01.domain).
  2015-02-24 17:48:01.826 32199 INFO nova.virt.libvirt.driver [-] [instance: 210f6191-a646-4bba-96ef-80df63d2df01] Instance destroyed successfully.
  2015-02-24 17:48:01.928 32199 ERROR nova.openstack.common.threadgroup [-] 'NoneType' object has no attribute '__getitem__'
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 125, in wait
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     x.wait()
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 47, in wait
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 173, in wait
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 121, in wait
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 293, in switch
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 212, in main
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 492, in run_service
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     service.start()
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     self.manager.init_host()
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1137, in init_host
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     self._destroy_evacuated_instances(context)
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 768, in _destroy_evacuated_instances
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     bdi, destroy_disks)
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1056, in destroy
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     destroy_disks, migrate_data)
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1164, in cleanup
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     instance=instance)
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 82, in __exit__
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     six.reraise(self.type_, self.value, self.tb)
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1153, in cleanup
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     self._disconnect_volume(connection_info, disk_dev)
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1334, in _disconnect_volume
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     return driver.disconnect_volume(connection_info, disk_dev)
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 272, in inner
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 1043, in disconnect_volume
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup     devices = mdev_info['devices']
  2015-02-24 17:48:01.928 32199 TRACE nova.openstack.common.threadgroup TypeError: 'NoneType' object has no attribute '__getitem__'

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1424968/+subscriptions
References

[Bug 1424968] [NEW] unhandled return from linuxscsi.find_multipath_device in disconnect_volume
From: Stephen Marron, 2015-02-24