yahoo-eng-team team mailing list archive

[Lee Yarwood <lyarwood@xxxxxxxxxx>]

> 1. [version] My environment is actually Kilo (openstack-nova-compute-2015.1.1-1.el7.noarch),
> but I belive the implementation is not much 

Actually it has, os-brick now handles disconnect_volume calls for
multipath iSCSI volumes. The code within os-brick is also being heavily
refactored at the moment on the topic below :

https://review.openstack.org/#/q/status:open+project:openstack/os-
brick+branch:master+topic:refactor/multipath

Marking this bug as invalid for now, if you can reproduce against a
supported release please add os-brick to the bug.

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1686868

Title:
  live-migration failed when multipath command returned code 1

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  ### Steps to reproduce
  I have two compute node, source com and dest com.
  I did execute "live migration" command.
    $ nova live-migration <instance id> <dest com>

  ### Expected result
  The instance is migrated to dest com host and the status change from MIGRATING to ACTIVE.

  ### Actual result
  During _post_livemigration function, the multipath command somehow returned "code 1" , I think it's a temporary error, on source com.
  And the status remains MIGRATING

  I think retrying multipath command can avoid this problem.

  ### Environment
  1. [version] My environment is actually Kilo (openstack-nova-compute-2015.1.1-1.el7.noarch), but I belive the implementation is not much changed on the latest version.
  2. [hypervisor] Libvirt + KVM (RHEL7.1)
  3. [storage] FC Multipath
  4. [network] openvswitch vlan

  ### Logs 
  nova-compute.log on source com:

  Lock "connect_volume" acquired by "disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:444
  Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123 execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199
  CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123" returned: 1 in 0.109s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225
  u'sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:258
  Multipath call failed exit (1)
  Lock "connect_volume" released by "disconnect_volume" :: held 0.112s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:456
  [instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Error monitoring migration: 'NoneType' object has no attribute '__getitem__'
   Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5708, in _live_migration
      dom, finish_event)
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5638, in _live_migration_monitor
      migrate_data)
    File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
      payload)
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
      return f(self, context, *args, **kw)
    File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 355, in decorated_function
      kwargs['instance'], e, sys.exc_info())
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 343, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5355, in _post_live_migration
      migrate_data)
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5983, in post_live_migration
      self._disconnect_volume(connection_info, disk_dev)
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1023, in _disconnect_volume
      return driver.disconnect_volume(connection_info, disk_dev)
    File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
      return f(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 1402, in disconnect_volume
      devices = mdev_info['devices']
  TypeError: 'NoneType' object has no attribute '__getitem__'
  [instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Live migration monitoring is all done _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:5715

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1686868/+subscriptions