← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #46819

[Bug 1549074] [NEW] Liberty - Unexpected exception in API method

  Thread Previous • Date Previous • • Thread Next 
Public bug reported:

This is 3 Node Liberty setup (controller, Network and Compute) configured with neutron Network option 2 : Self servicing network and is with openvswitch.
I found everything is working fine except while launching instance I'm getting error as follows , By tracing keystone.log and nova-api.log and based on time stamp I see Keystone fails the authentication followed by nova-api started throwing the exception errors.

Attached the config files and logs for you reference ... Request you to
help me ASAP.

Controller: 10.182.131.93
Network: 10.182.131.84
Compute: 10.182.131.87

Thank You

******************************************************
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] Unexpected exception in API method
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return self._plugin.get_auth_ref(session, **kwargs)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     authenticated=False, log=False, **rkwargs)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d2ea540d-cdbc-44b9-bbed-ab2fc92ebbe4)
2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions
2016-02-23 18:46:50.048 17814 INFO nova.api.openstack.wsgi [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneclient.exceptions.Unauthorized'>
******************************************************

** Affects: nova
     Importance: Undecided
         Status: New

** Patch added: "Config_and_logs.txt"
   https://bugs.launchpad.net/bugs/1549074/+attachment/4579555/+files/Config_and_logs.txt

** Description changed:

  This is 3 Node Liberty setup (controller, Network and Compute) configured with neutron Network option 2 : Self servicing network and is with openvswitch.
  I found everything is working fine except while launching instance I'm getting error as follows , By tracing keystone.log and nova-api.log and based on time stamp I see Keystone fails the authentication followed by nova-api started throwing the exception errors.
  
  Attached the config files and logs for you reference ... Request you to
  help me ASAP.
  
+ Controller: 10.182.131.93
+ Network: 10.182.131.84
+ Compute: 10.182.131.87
+ 
  Thank You
- 
  
  ******************************************************
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] Unexpected exception in API method
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     **create_kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return self._plugin.get_auth_ref(session, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     authenticated=False, log=False, **rkwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d2ea540d-cdbc-44b9-bbed-ab2fc92ebbe4)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions
  2016-02-23 18:46:50.048 17814 INFO nova.api.openstack.wsgi [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'keystoneclient.exceptions.Unauthorized'>
  ******************************************************

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1549074

Title:
  Liberty - Unexpected exception in API method

Status in OpenStack Compute (nova):
  New

Bug description:
  This is 3 Node Liberty setup (controller, Network and Compute) configured with neutron Network option 2 : Self servicing network and is with openvswitch.
  I found everything is working fine except while launching instance I'm getting error as follows , By tracing keystone.log and nova-api.log and based on time stamp I see Keystone fails the authentication followed by nova-api started throwing the exception errors.

  Attached the config files and logs for you reference ... Request you
  to help me ASAP.

  Controller: 10.182.131.93
  Network: 10.182.131.84
  Compute: 10.182.131.87

  Thank You

  ******************************************************
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] Unexpected exception in API method
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     **create_kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return self._plugin.get_auth_ref(session, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     authenticated=False, log=False, **rkwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-d2ea540d-cdbc-44b9-bbed-ab2fc92ebbe4)
  2016-02-23 18:46:50.042 17814 ERROR nova.api.openstack.extensions
  2016-02-23 18:46:50.048 17814 INFO nova.api.openstack.wsgi [req-61818a91-ddff-4d25-8855-0dcde336e6b0 1cd7f867708b46e0a795abad8a7a66c3 6e6b9259a526447fa4274442b81c3b55 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'keystoneclient.exceptions.Unauthorized'>
  ******************************************************

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1549074/+subscriptions

Follow ups

  Thread Previous • Date Previous • • Thread Next