yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1686868] [NEW] live-migration failed when multipath command returned code 1

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: TetsuoTSUJII <tsujii.tetsuo@xxxxxxxxxxxxxx>
Date: Fri, 28 Apr 2017 01:20:14 -0000
Reply-to: Bug 1686868 <1686868@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

### Steps to reproduce
I have two compute node, source com and dest com.
I did execute "live migration" command.
  $ nova live-migration <instance id> <dest com>

### Expected result
The instance is migrated to dest com host and the status change from MIGRATING to ACTIVE.

### Actual result
During _post_livemigration function, the multipath command somehow returned "code 1" , I think it's a temporary error, on source com.
And the status remains MIGRATING

I think retrying multipath command can avoid this problem.

### Environment
1. [version] My environment is actually Kilo (openstack-nova-compute-2015.1.1-1.el7.noarch), but I belive the implementation is not much changed on the latest version.
2. [hypervisor] Libvirt + KVM (RHEL7.1)
3. [storage] FC Multipath
4. [network] openvswitch vlan

### Logs 
nova-compute.log on source com:

Lock "connect_volume" acquired by "disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:444
Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123 execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199
CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123" returned: 1 in 0.109s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225
u'sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:258
Multipath call failed exit (1)
Lock "connect_volume" released by "disconnect_volume" :: held 0.112s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:456
[instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Error monitoring migration: 'NoneType' object has no attribute '__getitem__'
 Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5708, in _live_migration
    dom, finish_event)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5638, in _live_migration_monitor
    migrate_data)
  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
    payload)
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
    return f(self, context, *args, **kw)
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 355, in decorated_function
    kwargs['instance'], e, sys.exc_info())
  File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 343, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5355, in _post_live_migration
    migrate_data)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5983, in post_live_migration
    self._disconnect_volume(connection_info, disk_dev)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1023, in _disconnect_volume
    return driver.disconnect_volume(connection_info, disk_dev)
  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 1402, in disconnect_volume
    devices = mdev_info['devices']
TypeError: 'NoneType' object has no attribute '__getitem__'
[instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Live migration monitoring is all done _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:5715

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1686868

Title:
  live-migration failed when multipath command returned code 1

Status in OpenStack Compute (nova):
  New

Bug description:
  ### Steps to reproduce
  I have two compute node, source com and dest com.
  I did execute "live migration" command.
    $ nova live-migration <instance id> <dest com>

  ### Expected result
  The instance is migrated to dest com host and the status change from MIGRATING to ACTIVE.

  ### Actual result
  During _post_livemigration function, the multipath command somehow returned "code 1" , I think it's a temporary error, on source com.
  And the status remains MIGRATING

  I think retrying multipath command can avoid this problem.

  ### Environment
  1. [version] My environment is actually Kilo (openstack-nova-compute-2015.1.1-1.el7.noarch), but I belive the implementation is not much changed on the latest version.
  2. [hypervisor] Libvirt + KVM (RHEL7.1)
  3. [storage] FC Multipath
  4. [network] openvswitch vlan

  ### Logs 
  nova-compute.log on source com:

  Lock "connect_volume" acquired by "disconnect_volume" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:444
  Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123 execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:199
  CMD "sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123" returned: 1 in 0.109s execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:225
  u'sudo nova-rootwrap /etc/nova/rootwrap.conf multipath -l 123456789012345678901234567890123' failed. Not Retrying. execute /usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py:258
  Multipath call failed exit (1)
  Lock "connect_volume" released by "disconnect_volume" :: held 0.112s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:456
  [instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Error monitoring migration: 'NoneType' object has no attribute '__getitem__'
   Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5708, in _live_migration
      dom, finish_event)
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5638, in _live_migration_monitor
      migrate_data)
    File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
      payload)
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
      return f(self, context, *args, **kw)
    File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 355, in decorated_function
      kwargs['instance'], e, sys.exc_info())
    File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
      six.reraise(self.type_, self.value, self.tb)
    File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 343, in decorated_function
      return function(self, context, *args, **kwargs)
    File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5355, in _post_live_migration
      migrate_data)
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5983, in post_live_migration
      self._disconnect_volume(connection_info, disk_dev)
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1023, in _disconnect_volume
      return driver.disconnect_volume(connection_info, disk_dev)
    File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 445, in inner
      return f(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/volume.py", line 1402, in disconnect_volume
      devices = mdev_info['devices']
  TypeError: 'NoneType' object has no attribute '__getitem__'
  [instance: abcdefgh-ijkl-mnlo-pqrs-tuvwxyz12345] Live migration monitoring is all done _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py:5715

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1686868/+subscriptions
Follow ups

[Bug 1686868] Re: live-migration failed when multipath command returned code 1
From: Lee Yarwood, 2017-05-07