yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1675720] [NEW] NOVA report "Unexpected API Error" on Newton while run "Nova list"

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: "ming.yang" <1675720@xxxxxxxxxxxxxxxxxx>
Date: Fri, 24 Mar 2017 10:28:46 -0000
Reply-to: Bug 1675720 <1675720@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Description
===========
While run "nova list" to get instance list, NOVA report "[root@ARCHS22K8144 httpd]# nova list
ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-e08c4a96-d986-4250-8186-7d5974b0a1b5)" randomly.

Steps to reproduce
==================
for i in `seq 0 100`; do echo ===========================$i; nova list; sleep 3; done

Expected result
===============
it should list all instances successfully.

Actual result
=============
most time it can list instances, but some time it report "Unexpected API Error".

Environment
===========
1. Exact version of OpenStack you are running. See the following
  list for all releases: http://docs.openstack.org/releases/
# rpm -qa | grep nova
openstack-nova-common-14.0.3-1.el7.noarch
openstack-nova-console-14.0.3-1.el7.noarch
python2-novaclient-6.0.0-1.el7.noarch
openstack-nova-novncproxy-14.0.3-1.el7.noarch
python-nova-14.0.3-1.el7.noarch
openstack-nova-api-14.0.3-1.el7.noarch
openstack-nova-conductor-14.0.3-1.el7.noarch
openstack-nova-scheduler-14.0.3-1.el7.noarch

2. Which hypervisor did you use?
  Libvirt + KVM

2. Which storage type did you use?
   Ceph

3. Which networking type did you use?
   nova-network

Logs & Configs
==============
nova.conf:
[DEFAULT]
auth_strategy=keystone
my_ip=23.10.10.4
use_neutron=true
scheduler_default_filters=AllHostsFilter
enabled_apis=osapi_compute,metadata
firewall_driver=nova.virt.firewall.NoopFirewallDriver
debug=true
verbose=true
rpc_backend=rabbit
[api_database]
connection=mysql+pymysql://nova:open1sys@controller/nova_api
[cinder]
os_region_name=RegionOne
[database]
connection=mysql+pymysql://nova:open1sys@controller/nova
[glance]
api_servers=http://23.10.10.254:9292
[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = 23.10.10.4:11211,23.10.10.5:11211,23.10.10.6:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = open1sys
[neutron]
url = http://controller:9696
auth_url = http://controller:35357
auth_type = password
project_domain_name = default
user_domain_name = default
region_name = RegionOne
project_name = service
username = neutron
password = open1sys
service_metadata_proxy = True
metadata_proxy_shared_secret = open1sys
[oslo_concurrency]
lock_path=/var/lib/nova/tmp
[oslo_messaging_rabbit]
rabbit_port=5672
rabbit_hosts=23.10.10.4:5672,23.10.10.5:5672,23.10.10.6:5672
rabbit_userid=openstack
rabbit_password=open1sys
[vnc]
vncserver_listen=$my_ip
vncserver_proxyclient_address=$my_ip

nova-api.log:
2017-03-24 03:14:54.618 3460 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
2017-03-24 03:14:54.619 3460 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0018852
2017-03-24 03:14:56.620 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42466) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
2017-03-24 03:14:56.623 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
2017-03-24 03:14:56.624 3452 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0018470
2017-03-24 03:14:56.776 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42472) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
2017-03-24 03:14:57.558 3452 INFO nova.osapi_compute.wsgi.server [req-2f8b6a98-a927-4073-8322-c73c282078f2 - - - - -] 23.10.10.1 "GET /v2.1/b4c6cddfe2f74046a33784d01ded0942 HTTP/1.1" status: 404 len: 331 time: 0.7801268
2017-03-24 03:14:57.572 3452 DEBUG nova.api.openstack.wsgi [req-8d59ffbe-f6b2-4ed3-aa08-cd9ac9f0e401 - - - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7a978d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
2017-03-24 03:14:57.573 3452 INFO nova.osapi_compute.wsgi.server [req-8d59ffbe-f6b2-4ed3-aa08-cd9ac9f0e401 - - - - -] 23.10.10.1 "GET /v2.1/ HTTP/1.1" status: 200 len: 717 time: 0.0126722
2017-03-24 03:14:57.950 3452 DEBUG nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x794c3d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
2017-03-24 03:14:57.956 3452 DEBUG nova.compute.api [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Searching by: {'deleted': False, 'project_id': u'b4c6cddfe2f74046a33784d01ded0942'} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2245
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Unexpected exception in API method
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 207,
in detail
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=True)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 347,
in _get_servers
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     sort_keys=sort_keys, sort_dirs=sort_dirs)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2313, in get_all
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     sort_dirs=sort_dirs)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 324, in get_by_filters
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     build_requests = cls.get_all(context)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 239, in get_all
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     db_build_reqs = cls._get_all_from_db(context)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 894, in
wrapper
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return fn(*args, **kwargs)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 234, in _get_all_from_db
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     db_reqs = query.all()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2588, in all
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return list(self)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self._execute_and_instances(context)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     close_with_result=True)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     **kw)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 905, in connection
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     execution_options=execution_options)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     engine, execution_options)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     conn = bind.contextual_connect()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     **kwargs)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.dispatch.engine_connect(self, self.__branch)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     fn(*args, **kw)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 80, in _connect_ping_listener
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     connection.scalar(select([1]))
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self.execute(object, *multiparams, **params).scalar()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return meth(self, multiparams, params)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return connection._execute_clauseelement(self, multiparams, params)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     compiled_sql, distilled_params
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     None, None)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self._autorollback()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self._root._rollback_impl()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 662, in _rollback_impl
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.dispatch.rollback(self)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     fn(*args, **kw)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/exc_filters.py", line 496, in pop_exc_tx
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     conn.info.pop(ROLLBACK_CAUSE_KEY, None)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 425, in info
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self.connection.info
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 318, in connection
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self._handle_dbapi_exception(e, None, None, None, None)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     exc_info
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 316, in connection
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self._revalidate_connection()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 393, in _revalidate_connection
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.__connection = self.engine.raw_connection(_connection=self)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.pool.unique_connection, _connection)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2080, in _wrap_pool_connect
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     util.reraise(*sys.exc_info())
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return fn()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return _ConnectionFairy._checkout(self)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     fairy = _ConnectionRecord.checkout(pool)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in checkout
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     rec.checkin()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     compat.reraise(exc_type, exc_value, exc_tb)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in checkout
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     dbapi_connection = rec.get_connection()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in get_connection
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.connection = self.__connect()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     connection = self.__pool._invoke_creator(self)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return dialect.connect(*cargs, **cparams)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self.dbapi.connect(*cargs, **cparams)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return Connection(*args, **kwargs)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 694, in __init__
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.connect()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 916, in connect
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self._request_authentication()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1124, in _request_authentication
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     auth_packet = self._read_packet()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     packet.check_error()
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     err.raise_mysql_exception(self._data)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     raise errorclass(errno, errval)
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions
2017-03-24 03:14:57.972 3452 INFO nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'>
2017-03-24 03:14:57.973 3452 DEBUG nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
<class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1044
2017-03-24 03:14:57.975 3452 INFO nova.osapi_compute.wsgi.server [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] 23.10.10.1 "GET /v2.1/b4c6cddfe2f74046a33784d01ded0942/servers/detail HTTP/1.1" status: 500 len: 574 time: 0.0400600
2017-03-24 03:14:58.626 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42502) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
2017-03-24 03:14:58.629 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
2017-03-24 03:14:58.631 3452 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0024850
2017-03-24 03:15:00.632 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42530) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
2017-03-24 03:15:00.634 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636

** Affects: nova
     Importance: Undecided
         Status: New

** Attachment added: "sosreport-ARCHS22K8144.tuc.stglabs.ibm.com-20170324032503.tar.xz"
   https://bugs.launchpad.net/bugs/1675720/+attachment/4844178/+files/sosreport-ARCHS22K8144.tuc.stglabs.ibm.com-20170324032503.tar.xz

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1675720

Title:
  NOVA report "Unexpected API Error" on Newton while run "Nova list"

Status in OpenStack Compute (nova):
  New

Bug description:
  Description
  ===========
  While run "nova list" to get instance list, NOVA report "[root@ARCHS22K8144 httpd]# nova list
  ERROR (ClientException): Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-e08c4a96-d986-4250-8186-7d5974b0a1b5)" randomly.

  Steps to reproduce
  ==================
  for i in `seq 0 100`; do echo ===========================$i; nova list; sleep 3; done

  Expected result
  ===============
  it should list all instances successfully.

  Actual result
  =============
  most time it can list instances, but some time it report "Unexpected API Error".

  Environment
  ===========
  1. Exact version of OpenStack you are running. See the following
    list for all releases: http://docs.openstack.org/releases/
  # rpm -qa | grep nova
  openstack-nova-common-14.0.3-1.el7.noarch
  openstack-nova-console-14.0.3-1.el7.noarch
  python2-novaclient-6.0.0-1.el7.noarch
  openstack-nova-novncproxy-14.0.3-1.el7.noarch
  python-nova-14.0.3-1.el7.noarch
  openstack-nova-api-14.0.3-1.el7.noarch
  openstack-nova-conductor-14.0.3-1.el7.noarch
  openstack-nova-scheduler-14.0.3-1.el7.noarch

  2. Which hypervisor did you use?
    Libvirt + KVM

  2. Which storage type did you use?
     Ceph

  3. Which networking type did you use?
     nova-network

  Logs & Configs
  ==============
  nova.conf:
  [DEFAULT]
  auth_strategy=keystone
  my_ip=23.10.10.4
  use_neutron=true
  scheduler_default_filters=AllHostsFilter
  enabled_apis=osapi_compute,metadata
  firewall_driver=nova.virt.firewall.NoopFirewallDriver
  debug=true
  verbose=true
  rpc_backend=rabbit
  [api_database]
  connection=mysql+pymysql://nova:open1sys@controller/nova_api
  [cinder]
  os_region_name=RegionOne
  [database]
  connection=mysql+pymysql://nova:open1sys@controller/nova
  [glance]
  api_servers=http://23.10.10.254:9292
  [keystone_authtoken]
  auth_uri = http://controller:5000
  auth_url = http://controller:35357
  memcached_servers = 23.10.10.4:11211,23.10.10.5:11211,23.10.10.6:11211
  auth_type = password
  project_domain_name = default
  user_domain_name = default
  project_name = service
  username = nova
  password = open1sys
  [neutron]
  url = http://controller:9696
  auth_url = http://controller:35357
  auth_type = password
  project_domain_name = default
  user_domain_name = default
  region_name = RegionOne
  project_name = service
  username = neutron
  password = open1sys
  service_metadata_proxy = True
  metadata_proxy_shared_secret = open1sys
  [oslo_concurrency]
  lock_path=/var/lib/nova/tmp
  [oslo_messaging_rabbit]
  rabbit_port=5672
  rabbit_hosts=23.10.10.4:5672,23.10.10.5:5672,23.10.10.6:5672
  rabbit_userid=openstack
  rabbit_password=open1sys
  [vnc]
  vncserver_listen=$my_ip
  vncserver_proxyclient_address=$my_ip

  nova-api.log:
  2017-03-24 03:14:54.618 3460 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
  2017-03-24 03:14:54.619 3460 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0018852
  2017-03-24 03:14:56.620 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42466) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
  2017-03-24 03:14:56.623 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
  2017-03-24 03:14:56.624 3452 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0018470
  2017-03-24 03:14:56.776 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42472) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
  2017-03-24 03:14:57.558 3452 INFO nova.osapi_compute.wsgi.server [req-2f8b6a98-a927-4073-8322-c73c282078f2 - - - - -] 23.10.10.1 "GET /v2.1/b4c6cddfe2f74046a33784d01ded0942 HTTP/1.1" status: 404 len: 331 time: 0.7801268
  2017-03-24 03:14:57.572 3452 DEBUG nova.api.openstack.wsgi [req-8d59ffbe-f6b2-4ed3-aa08-cd9ac9f0e401 - - - - -] Calling method '<bound method VersionsController.show of <nova.api.openstack.compute.versionsV21.VersionsController object at 0x7a978d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
  2017-03-24 03:14:57.573 3452 INFO nova.osapi_compute.wsgi.server [req-8d59ffbe-f6b2-4ed3-aa08-cd9ac9f0e401 - - - - -] 23.10.10.1 "GET /v2.1/ HTTP/1.1" status: 200 len: 717 time: 0.0126722
  2017-03-24 03:14:57.950 3452 DEBUG nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Calling method '<bound method ServersController.detail of <nova.api.openstack.compute.servers.ServersController object at 0x794c3d0>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
  2017-03-24 03:14:57.956 3452 DEBUG nova.compute.api [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Searching by: {'deleted': False, 'project_id': u'b4c6cddfe2f74046a33784d01ded0942'} get_all /usr/lib/python2.7/site-packages/nova/compute/api.py:2245
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Unexpected exception in API method
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 207,
  in detail
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=True)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 347,
  in _get_servers
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     sort_keys=sort_keys, sort_dirs=sort_dirs)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 2313, in get_all
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     sort_dirs=sort_dirs)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 324, in get_by_filters
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     build_requests = cls.get_all(context)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 239, in get_all
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     db_build_reqs = cls._get_all_from_db(context)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 894, in
  wrapper
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return fn(*args, **kwargs)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/build_request.py", line 234, in _get_all_from_db
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     db_reqs = query.all()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2588, in all
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return list(self)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self._execute_and_instances(context)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     close_with_result=True)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     **kw)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 905, in connection
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     execution_options=execution_options)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     engine, execution_options)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     conn = bind.contextual_connect()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     **kwargs)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.dispatch.engine_connect(self, self.__branch)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     fn(*args, **kw)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 80, in _connect_ping_listener
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     connection.scalar(select([1]))
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self.execute(object, *multiparams, **params).scalar()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return meth(self, multiparams, params)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return connection._execute_clauseelement(self, multiparams, params)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     compiled_sql, distilled_params
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1078, in _execute_context
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     None, None)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self._autorollback()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self._root._rollback_impl()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 662, in _rollback_impl
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.dispatch.rollback(self)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     fn(*args, **kw)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/exc_filters.py", line 496, in pop_exc_tx
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     conn.info.pop(ROLLBACK_CAUSE_KEY, None)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 425, in info
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self.connection.info
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 318, in connection
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self._handle_dbapi_exception(e, None, None, None, None)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     exc_info
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 316, in connection
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self._revalidate_connection()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 393, in _revalidate_connection
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.__connection = self.engine.raw_connection(_connection=self)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.pool.unique_connection, _connection)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2080, in _wrap_pool_connect
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     util.reraise(*sys.exc_info())
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return fn()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in unique_connection
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return _ConnectionFairy._checkout(self)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     fairy = _ConnectionRecord.checkout(pool)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 485, in checkout
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     rec.checkin()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     compat.reraise(exc_type, exc_value, exc_tb)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 482, in checkout
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     dbapi_connection = rec.get_connection()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 563, in get_connection
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.connection = self.__connect()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     connection = self.__pool._invoke_creator(self)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return dialect.connect(*cargs, **cparams)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return self.dbapi.connect(*cargs, **cparams)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     return Connection(*args, **kwargs)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 694, in __init__
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self.connect()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 916, in connect
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     self._request_authentication()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1124, in _request_authentication
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     auth_packet = self._read_packet()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     packet.check_error()
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     err.raise_mysql_exception(self._data)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions     raise errorclass(errno, errval)
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
  2017-03-24 03:14:57.967 3452 ERROR nova.api.openstack.extensions
  2017-03-24 03:14:57.972 3452 INFO nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'>
  2017-03-24 03:14:57.973 3452 DEBUG nova.api.openstack.wsgi [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> __call__ /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:1044
  2017-03-24 03:14:57.975 3452 INFO nova.osapi_compute.wsgi.server [req-e08c4a96-d986-4250-8186-7d5974b0a1b5 - - - - -] 23.10.10.1 "GET /v2.1/b4c6cddfe2f74046a33784d01ded0942/servers/detail HTTP/1.1" status: 500 len: 574 time: 0.0400600
  2017-03-24 03:14:58.626 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42502) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
  2017-03-24 03:14:58.629 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636
  2017-03-24 03:14:58.631 3452 INFO nova.osapi_compute.wsgi.server [-] 23.10.10.1 "OPTIONS / HTTP/1.0" status: 200 len: 497 time: 0.0024850
  2017-03-24 03:15:00.632 3452 DEBUG nova.osapi_compute.wsgi.server [req-c89693a9-2ee6-4e90-9341-a41b19ebbee6 - - - - -] (3452) accepted ('23.10.10.1', 42530) server /usr/lib/python2.7/site-packages/eventlet/wsgi.py:867
  2017-03-24 03:15:00.634 3452 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x57a3c10>>' _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:636

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1675720/+subscriptions
Follow ups

[Bug 1675720] Re: NOVA report "Unexpected API Error" on Newton while run "Nova list"
From: jichenjc, 2017-03-27