← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #56762

[Bug 1625321] [NEW] Hitting "Authentication required" on migration when building network info model

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

Description
===========

I observed this issue a few times, mainly while doing stress testing of
cold migration.

Neutron returns this error (see stack trace):
Unauthorized: Authentication required

Steps to reproduce
==================

1) Boot multiple vms (5-6) 
2) Stress test cold migration: migrate all vms, once confirmed, redo again, over and over 

Actual result
=============

2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] Traceback (most recent call last): 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4271, in finish_resize 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] disk_info, image_meta) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4210, in _finish_resize 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] network_info = self.network_api.get_instance_nw_info(context, instance) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 253, in get_instance_nw_info 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] result = self._get_instance_nw_info(context, instance, **kwargs) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 994, in _get_instance_nw_info 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] preexisting_port_ids) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1787, in _build_network_info_model 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] context, instance, networks, port_ids) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1017, in _gather_port_ids_and_networks 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] net_ids) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 207, in _get_available_networks 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] nets = neutron.list_networks(**search_opts).get('networks', []) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 105, in with_params 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] ret = self.function(instance, *args, **kwargs) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 730, in list_networks 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] **_params) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in list 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] for r in self._pagination(collection, path, **params): 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 399, in _pagination 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] res = self.get(path, params=params) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in get 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] headers=headers, params=params) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 346, in retry_request 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] headers=headers, params=params) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 309, in do_request 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] self._handle_fault_response(status_code, replybody, resp) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 284, in _handle_fault_response 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] exception_handler_v20(status_code, error_body) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] request_ids=request_ids) 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] Unauthorized: Authentication required 
2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] Neutron server returns request_ids: ['req-ad48fbe3-8405-4393-9251-ca5fa20c9653'] 

Environment
===========

Mitaka

** Affects: nova
     Importance: Undecided
         Status: Incomplete


** Tags: neutron

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1625321

Title:
  Hitting "Authentication required" on migration when building network
  info model

Status in OpenStack Compute (nova):
  Incomplete

Bug description:
  Description
  ===========

  I observed this issue a few times, mainly while doing stress testing
  of cold migration.

  Neutron returns this error (see stack trace):
  Unauthorized: Authentication required

  Steps to reproduce
  ==================

  1) Boot multiple vms (5-6) 
  2) Stress test cold migration: migrate all vms, once confirmed, redo again, over and over 

  Actual result
  =============

  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] Traceback (most recent call last): 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4271, in finish_resize 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] disk_info, image_meta) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4210, in _finish_resize 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] network_info = self.network_api.get_instance_nw_info(context, instance) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 253, in get_instance_nw_info 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] result = self._get_instance_nw_info(context, instance, **kwargs) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 994, in _get_instance_nw_info 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] preexisting_port_ids) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1787, in _build_network_info_model 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] context, instance, networks, port_ids) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1017, in _gather_port_ids_and_networks 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] net_ids) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 207, in _get_available_networks 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] nets = neutron.list_networks(**search_opts).get('networks', []) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 105, in with_params 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] ret = self.function(instance, *args, **kwargs) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 730, in list_networks 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] **_params) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 384, in list 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] for r in self._pagination(collection, path, **params): 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 399, in _pagination 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] res = self.get(path, params=params) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 369, in get 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] headers=headers, params=params) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 346, in retry_request 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] headers=headers, params=params) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 309, in do_request 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] self._handle_fault_response(status_code, replybody, resp) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 284, in _handle_fault_response 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] exception_handler_v20(status_code, error_body) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 92, in exception_handler_v20 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] request_ids=request_ids) 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] Unauthorized: Authentication required 
  2016-09-02 13:53:14.887 61968 ERROR nova.compute.manager [instance: da6e6dd9-6042-478f-9886-c88d332eb4e3] Neutron server returns request_ids: ['req-ad48fbe3-8405-4393-9251-ca5fa20c9653'] 

  Environment
  ===========

  Mitaka

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1625321/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next