yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1599057] Re: live-migration Interrupt when get network info failed

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Timofey Durakov <tdurakov@xxxxxxxxxxxx>
Date: Tue, 17 Jan 2017 21:10:33 -0000
Reply-to: Bug 1599057 <1599057@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
*** This bug is a duplicate of bug 1647451 ***
    https://bugs.launchpad.net/bugs/1647451

this is actually the duplicate for 1647451

** This bug has been marked a duplicate of bug 1647451
   Post live migration step could fail due to auth errors

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1599057

Title:
  live-migration Interrupt when get network info failed

Status in OpenStack Compute (nova):
  In Progress

Bug description:
  Description
  ===========
  When an exception raised by neutron，live migrate will be interrupt and vm task_state keep in migrating.

  Steps to reproduce
  ==================
  1.Set the Token timeout after 5 minutes.

  2.Login dashboard, and live-migrate an CentOS instance after 4
  minutes(the token will timeout after 1min)

  3.bug reproduced,because of neutronclient  Authentication required
  (token timeout)

  Expected result
  ===============
  Instance rollback or directly in error state.

  Actual result
  =============
  Live migrate operation stoped，but instance task_state keeping in "migrating"

  Environment
  ===========
  version: Mitaka

  hypervisor : Libvirt + KVM

  storage: LVM

  networking type: Neutron with OpenVSwitch

  
  Logs & Configs
  ==============
  In source node,nova-compute log as follow:

  2016-07-02 16:14:17.809 10162 INFO nova.compute.manager [req-d322c8ac-2d6d-4960-8275-ac6e3f26f7ac ebe821cc991f4657aa3002054739933c 71acb857b6e34df6bfa2da07b0ce7902 - - -] [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] _post_live_migration() is started..
  2016-07-02 16:14:18.072 10162 WARNING nova.virt.libvirt.driver [req-d322c8ac-2d6d-4960-8275-ac6e3f26f7ac ebe821cc991f4657aa3002054739933c 71acb857b6e34df6bfa2da07b0ce7902 - - -] [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] Error monitoring migration: Authentication required
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] Traceback (most recent call last):
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6504, in _live_migration
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     dom, finish_event)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6434, in _live_migration_monitor
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     migrate_data)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     payload)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     six.reraise(self.type_, self.value, self.tb)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     return f(self, context, *args, **kw)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 405, in decorated_function
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     kwargs['instance'], e, sys.exc_info())
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     six.reraise(self.type_, self.value, self.tb)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     return function(self, context, *args, **kwargs)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 6329, in _post_live_migration
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     network_info = self._get_instance_nw_info(ctxt, instance)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1647, in _get_instance_nw_info
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     return self.network_api.get_instance_nw_info(context, instance)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 767, in get_instance_nw_info
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     preexisting_port_ids)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 783, in _get_instance_nw_info
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     preexisting_port_ids)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1741, in _rebuild_network_info_model
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     context, instance, current_neutron_ports)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1701, in _gather_networks_by_interface
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     iface['network_id'])
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 251, in _get_available_networks
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     nets = neutron.list_networks(**search_opts).get('networks', [])
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 99, in with_params
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     ret = self.function(instance, *args, **kwargs)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 526, in list_networks
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     **_params)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 304, in list
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     for r in self._pagination(collection, path, **params):
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 317, in _pagination
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     res = self.get(path, params=params)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 290, in get
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     headers=headers, params=params)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 267, in retry_request
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     headers=headers, params=params)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 208, in do_request
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     self._handle_fault_response(status_code, replybody)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     exception_handler_v20(status_code, des_error_body)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 80, in exception_handler_v20
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8]     message=message)
  2016-07-02 16:14:18.072 10162 TRACE nova.virt.libvirt.driver [instance: 37d55e32-751a-415a-b707-4f54bc06b6b8] NeutronClientException: Authentication required

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1599057/+subscriptions
References

[Bug 1599057] [NEW] live-migration Interrupt when get network info failed
From: Zengzhi Wang, 2016-07-05