yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1534273] [NEW] Nova - Unexpected API Error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ahmed <aali0101@xxxxxxxxx>
Date: Thu, 14 Jan 2016 19:21:52 -0000
Reply-to: Bug 1534273 <1534273@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Hi,

Launching new/first instance fails with an error, this is a  new openstack Liberty with 2 node deployment, following official documentation:
http://docs.openstack.org/liberty/install-guide-rdo/launch-instance-public.html


Command used::
[root@xepcloud ~]# nova boot --flavor m1.tiny --image cirros --nic net-id=8fb32974-8dcf-47c8-a42b-a890e47725f4 --security-group default --key-name mykey public-instance

Error::
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API 
log if possible.
<class 'keystoneclient.exceptions.BadRequest'> (HTTP 500) (Request-ID: req-a2d56513-0c19-46e8-9a45-d63fcee17224)

Note:
Could this be related to networking config? I was not sure what IP to use for OVERLAY_INTERFACE_IP_ADDRESS, so I used mgmt IP but I have a second physical interface for public access with no ip assigned.


nova API logs:


2016-01-14 11:00:13.292 11528 INFO nova.osapi_compute.wsgi.server [req-c3f11e04-e83e-46c8-9f3b-7a1709f5cc5d 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "GET /v2/ee13d79dc9954b458a8d0f173bd63ccb/flavors?is_public=None HTTP/1.1" status: 200 len: 1477 time: 0.0188160
2016-01-14 11:00:13.311 11528 INFO nova.osapi_compute.wsgi.server [req-696093c4-6513-4209-9f56-4a0ae9488a25 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "GET /v2/ee13d79dc9954b458a8d0f173bd63ccb/flavors/1 HTTP/1.1" status: 200 len: 629 time: 0.0156209
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions [req-a2d56513-0c19-46e8-9a45-d63fcee17224 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] Unexpected exception in API method
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     **create_kwargs)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     authenticated=False, log=False)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions BadRequest: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-5a787291-5ea7-42ec-8afc-c9cb2e826fa8)
2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions
2016-01-14 11:00:16.151 11528 INFO nova.api.openstack.wsgi [req-a2d56513-0c19-46e8-9a45-d63fcee17224 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'keystoneclient.exceptions.BadRequest'>
2016-01-14 11:00:16.154 11528 INFO nova.osapi_compute.wsgi.server [req-a2d56513-0c19-46e8-9a45-d63fcee17224 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "POST /v2/ee13d79dc9954b458a8d0f173bd63ccb/servers HTTP/1.1" status: 500 len: 441 time: 2.8398080
2016-01-14 11:00:47.647 11521 INFO nova.osapi_compute.wsgi.server [req-194d248a-5394-4cf9-bd38-6292c82ed221 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "GET /v2/ HTTP/1.1" status: 200 len: 579 time: 0.6176820
2016-01-14 11:00:48.266 11521 INFO nova.osapi_compute.wsgi.server [req-6f5d1860-e3e7-490c-b68a-98f334beba14 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "GET /v2/ee13d79dc9954b458a8d0f173bd63ccb/servers/detail HTTP/1.1" status: 200 len: 211 time: 0.4220510

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1534273

Title:
  Nova - Unexpected API Error

Status in OpenStack Compute (nova):
  New

Bug description:
  Hi,

  Launching new/first instance fails with an error, this is a  new openstack Liberty with 2 node deployment, following official documentation:
  http://docs.openstack.org/liberty/install-guide-rdo/launch-instance-public.html

  
  Command used::
  [root@xepcloud ~]# nova boot --flavor m1.tiny --image cirros --nic net-id=8fb32974-8dcf-47c8-a42b-a890e47725f4 --security-group default --key-name mykey public-instance

  Error::
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API 
  log if possible.
  <class 'keystoneclient.exceptions.BadRequest'> (HTTP 500) (Request-ID: req-a2d56513-0c19-46e8-9a45-d63fcee17224)

  Note:
  Could this be related to networking config? I was not sure what IP to use for OVERLAY_INTERFACE_IP_ADDRESS, so I used mgmt IP but I have a second physical interface for public access with no ip assigned.

  
  nova API logs:

  
  2016-01-14 11:00:13.292 11528 INFO nova.osapi_compute.wsgi.server [req-c3f11e04-e83e-46c8-9f3b-7a1709f5cc5d 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "GET /v2/ee13d79dc9954b458a8d0f173bd63ccb/flavors?is_public=None HTTP/1.1" status: 200 len: 1477 time: 0.0188160
  2016-01-14 11:00:13.311 11528 INFO nova.osapi_compute.wsgi.server [req-696093c4-6513-4209-9f56-4a0ae9488a25 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "GET /v2/ee13d79dc9954b458a8d0f173bd63ccb/flavors/1 HTTP/1.1" status: 200 len: 629 time: 0.0156209
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions [req-a2d56513-0c19-46e8-9a45-d63fcee17224 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] Unexpected exception in API method
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 73, in wrapper
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 611, in create
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     **create_kwargs)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 149, in inner
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     rv = f(*args, **kwargs)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1581, in create
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     check_server_group_quota=check_server_group_quota)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 1181, in _create_instance
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     auto_disk_config, reservation_id, max_count)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 955, in _validate_and_build_base_options
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     pci_request_info, requested_networks)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1059, in create_pci_requests_for_sriov_ports
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     neutron = get_client(context, admin=True)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 237, in get_client
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     auth_token = _ADMIN_AUTH.get_token(_SESSION)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return self.get_access(session).auth_token
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     self.auth_ref = self.get_auth_ref(session)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py", line 88, in get_auth_ref
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     authenticated=False, log=False)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return self.request(url, 'POST', **kwargs)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions     raise exceptions.from_response(resp, method, url)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions BadRequest: Expecting to find username or userId in passwordCredentials - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-5a787291-5ea7-42ec-8afc-c9cb2e826fa8)
  2016-01-14 11:00:16.143 11528 ERROR nova.api.openstack.extensions
  2016-01-14 11:00:16.151 11528 INFO nova.api.openstack.wsgi [req-a2d56513-0c19-46e8-9a45-d63fcee17224 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'keystoneclient.exceptions.BadRequest'>
  2016-01-14 11:00:16.154 11528 INFO nova.osapi_compute.wsgi.server [req-a2d56513-0c19-46e8-9a45-d63fcee17224 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "POST /v2/ee13d79dc9954b458a8d0f173bd63ccb/servers HTTP/1.1" status: 500 len: 441 time: 2.8398080
  2016-01-14 11:00:47.647 11521 INFO nova.osapi_compute.wsgi.server [req-194d248a-5394-4cf9-bd38-6292c82ed221 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "GET /v2/ HTTP/1.1" status: 200 len: 579 time: 0.6176820
  2016-01-14 11:00:48.266 11521 INFO nova.osapi_compute.wsgi.server [req-6f5d1860-e3e7-490c-b68a-98f334beba14 7d4aa0f0645248f3b49ec2a4956a2535 ee13d79dc9954b458a8d0f173bd63ccb - - -] 192.168.178.90 "GET /v2/ee13d79dc9954b458a8d0f173bd63ccb/servers/detail HTTP/1.1" status: 200 len: 211 time: 0.4220510

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1534273/+subscriptions
Follow ups

[Bug 1534273] Re: Keystone configuration options for nova.conf missing from Redhat/CentOS install guide
From: Matt Kassawara, 2016-02-08
[Bug 1534273] Re: Nova - Unexpected API Error
From: Augustina Ragwitz, 2016-01-15