yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1404764] [NEW] share storage live migration fails at _post_live_migration() function, but the status of this instance is still "migrating".

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Rong Han ZTE <spring068@xxxxxxx>
Date: Mon, 22 Dec 2014 01:03:57 -0000
Reply-to: Bug 1404764 <1404764@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

A share storage live-migration failed at function of "
_post_live_migration() " because  umount command failed, but the status
of this instance is still migrating.

Log is as follows:

2014-12-19 16:45:32.741 6127 INFO nova.compute.manager [-] [instance: e9fab51d-8e13-416b-b2c9-211e04ba35b2] _post_live_migration() is started..
2014-12-19 16:45:32.779 6127 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.47.158.165
2014-12-19 16:45:32.845 6127 INFO nova.compute.manager [-] [instance: e9fab51d-8e13-416b-b2c9-211e04ba35b2] During sync_power_state the instance has a pending task. Skip.
2014-12-19 16:45:33.041 6127 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     self.f(*self.args, **self.kw)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4987, in wait_for_live_migration
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     migrate_data)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     payload)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     six.reraise(self.type_, self.value, self.tb)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     return f(self, context, *args, **kw)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     e, sys.exc_info())
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     six.reraise(self.type_, self.value, self.tb)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 356, in decorated_function
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     return function(self, context, *args, **kwargs)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4826, in _post_live_migration
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     migrate_data)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5185, in post_live_migration
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     self._umount_instance_sysdisk(instance)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2214, in _umount_instance_sysdisk
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     utils.execute('umount', mount_path, run_as_root=True)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 165, in execute
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     return processutils.execute(*cmd, **kwargs)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 195, in execute
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     cmd=sanitized_cmd)
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall ProcessExecutionError: Unexpected error while running command.
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Command: sudo nova-rootwrap /etc/nova/rootwrap.conf umount /var/lib/nova/instances/e9fab51d-8e13-416b-b2c9-211e04ba35b2/sysdisk
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Exit code: 32
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Stdout: u''
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Stderr: u'umount: /var/lib/nova/instances/e9fab51d-8e13-416b-b2c9-211e04ba35b2/sysdisk: target is busy.\n        (In some cases useful info about processes that use\n         the device is found by lsof(8) or fuser(1))\n'
2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1404764

Title:
  share storage live migration  fails at _post_live_migration()
  function, but the status of this instance is still "migrating".

Status in OpenStack Compute (Nova):
  New

Bug description:
  A share storage live-migration failed at function of "
  _post_live_migration() " because  umount command failed, but the
  status of this instance is still migrating.

  Log is as follows:

  2014-12-19 16:45:32.741 6127 INFO nova.compute.manager [-] [instance: e9fab51d-8e13-416b-b2c9-211e04ba35b2] _post_live_migration() is started..
  2014-12-19 16:45:32.779 6127 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.47.158.165
  2014-12-19 16:45:32.845 6127 INFO nova.compute.manager [-] [instance: e9fab51d-8e13-416b-b2c9-211e04ba35b2] During sync_power_state the instance has a pending task. Skip.
  2014-12-19 16:45:33.041 6127 ERROR nova.openstack.common.loopingcall [-] in fixed duration looping call
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Traceback (most recent call last):
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/loopingcall.py", line 78, in _inner
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     self.f(*self.args, **self.kw)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4987, in wait_for_live_migration
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     migrate_data)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     payload)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     six.reraise(self.type_, self.value, self.tb)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     return f(self, context, *args, **kw)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 369, in decorated_function
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     e, sys.exc_info())
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     six.reraise(self.type_, self.value, self.tb)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 356, in decorated_function
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     return function(self, context, *args, **kwargs)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4826, in _post_live_migration
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     migrate_data)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 5185, in post_live_migration
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     self._umount_instance_sysdisk(instance)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2214, in _umount_instance_sysdisk
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     utils.execute('umount', mount_path, run_as_root=True)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/utils.py", line 165, in execute
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     return processutils.execute(*cmd, **kwargs)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall   File "/usr/lib/python2.7/site-packages/nova/openstack/common/processutils.py", line 195, in execute
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall     cmd=sanitized_cmd)
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall ProcessExecutionError: Unexpected error while running command.
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Command: sudo nova-rootwrap /etc/nova/rootwrap.conf umount /var/lib/nova/instances/e9fab51d-8e13-416b-b2c9-211e04ba35b2/sysdisk
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Exit code: 32
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Stdout: u''
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall Stderr: u'umount: /var/lib/nova/instances/e9fab51d-8e13-416b-b2c9-211e04ba35b2/sysdisk: target is busy.\n        (In some cases useful info about processes that use\n         the device is found by lsof(8) or fuser(1))\n'
  2014-12-19 16:45:33.041 6127 TRACE nova.openstack.common.loopingcall

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1404764/+subscriptions
Follow ups

[Bug 1404764] Re: share storage live migration fails at _post_live_migration() function, but the status of this instance is still "migrating".
From: Rong Han ZTE, 2014-12-23
[Bug 1404764] [NEW] share storage live migration fails at _post_live_migration() function, but the status of this instance is still "migrating".
From: Rong Han ZTE, 2014-12-22
References

[Bug 1404764] [NEW] share storage live migration fails at _post_live_migration() function, but the status of this instance is still "migrating".
From: Rong Han ZTE, 2014-12-22