yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1688220] Re: Operational error thrown by nova-api while serving the request

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Ashish Kumar Gupta <guptaas@xxxxxxxxxx>
Date: Fri, 05 May 2017 06:46:23 -0000
Reply-to: Bug 1688220 <1688220@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
@jichenjc .
Can you please tell me what configuration in mysql needed to solve this issue.

** Changed in: nova
       Status: Incomplete => Opinion

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1688220

Title:
  Operational error thrown by nova-api while serving the request

Status in OpenStack Compute (nova):
  Opinion

Bug description:
  Request to be served by nova fails with operational error:
  For reference adding floating ip to the server 
  Input task :
  Rally:
  "NovaServers.bootvm_userdata_test_snat": [
      {
        "runner": {
          "type": "constant", 
          "concurrency": 7, 
          "times": 22
        }, 

  Below traces can be found: req-d5bc233e-d1de-45d8-a353-c17ced17cd63

  2017-04-28 11:20:50.598 21334 DEBUG nova.api.openstack.wsgi [req-d5bc233e-d1de-45d8-a353-c17ced17cd63 c_rally_46c61b3d_hnwpef3T c_rally_46c61b3d_aYPvYjVl] Action: 'action', calling method: <bound method FloatingIPActionController._add_floating_ip of <nova.api.openstack.compute.floating_ips.FloatingIPActionController object at 0x7f0e421a8750>>, body: {"addFloatingIp": {"fixed_address": "1.2.44.3", "address": "61.26.96.145"}} _process_stack /opt/stack/nova/nova/api/openstack/wsgi.py:623
  2017-04-28 11:20:50.601 21334 DEBUG nova.compute.api [req-d5bc233e-d1de-45d8-a353-c17ced17cd63 c_rally_46c61b3d_hnwpef3T c_rally_46c61b3d_aYPvYjVl] [instance: c57cddc2-3aed-4c47-99b0-6f3ac16015b2] Fetching instance by UUID get /opt/stack/nova/nova/compute/api.py:2352
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions [req-d5bc233e-d1de-45d8-a353-c17ced17cd63 c_rally_46c61b3d_hnwpef3T c_rally_46c61b3d_aYPvYjVl] Unexpected exception in API method
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/extensions.py", line 338, in wrapped
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/validation/__init__.py", line 108, in wrapper
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/compute/floating_ips.py", line 223, in _add_floating_ip
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     expected_attrs=['flavor'])
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/api/openstack/common.py", line 481, in get_instance
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     expected_attrs=expected_attrs)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 2355, in get
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     expected_attrs)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/compute/api.py", line 2315, in _get_instance
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     context, instance_uuid, expected_attrs=expected_attrs)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/objects/instance.py", line 463, in get_by_uuid
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     use_slave=use_slave)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 235, in wrapper
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     with reader_mode.using(context):
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return self.gen.next()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 944, in _transaction_scope
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     allow_async=self._allow_async) as resource:
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return self.gen.next()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 558, in _session
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     bind=self.connection, mode=self.mode)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 317, in _create_session
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     self._start()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 403, in _start
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     engine_args, maker_args)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 427, in _setup_for_connection
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     sql_connection=sql_connection, **engine_kwargs)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 155, in create_engine
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     test_conn = _test_connection(engine, max_retries, retry_interval)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 331, in _test_connection
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return engine.connect()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2018, in connect
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return self._connection_cls(self, **kwargs)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 72, in __init__
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     if connection is not None else engine.raw_connection()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     self.pool.unique_connection, _connection)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     e, dialect, self)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     util.raise_from_cause(newraise, exc_info)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return fn()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 318, in unique_connection
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return _ConnectionFairy._checkout(self)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     fairy = _ConnectionRecord.checkout(pool)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     rec = pool._do_get()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     self._dec_overflow()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     compat.reraise(exc_type, exc_value, exc_tb)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return self._create_connection()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return _ConnectionRecord(self)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     self.connection = self.__connect()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     connection = self.__pool._invoke_creator(self)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return dialect.connect(*cargs, **cparams)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return self.dbapi.connect(*cargs, **cparams)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/pymysql/__init__.py", line 90, in Connect
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     return Connection(*args, **kwargs)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 688, in __init__
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     self.connect()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 905, in connect
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     self._get_server_information()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1207, in _get_server_information
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     packet = self._read_packet()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 981, in _read_packet
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     packet.check_error()
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 393, in check_error
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     err.raise_mysql_exception(self._data)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions   File "/usr/local/lib/python2.7/dist-packages/pymysql/err.py", line 107, in raise_mysql_exception
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions     raise errorclass(errno, errval)
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
  2017-04-28 11:20:50.609 21334 ERROR nova.api.openstack.extensions
  2017-04-28 11:20:50.613 21334 INFO nova.api.openstack.wsgi [req-d5bc233e-d1de-45d8-a353-c17ced17cd63 c_rally_46c61b3d_hnwpef3T c_rally_46c61b3d_aYPvYjVl] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'>
  2017-04-28 11:20:50.614 21334 DEBUG nova.api.openstack.wsgi [req-d5bc233e-d1de-45d8-a353-c17ced17cd63 c_rally_46c61b3d_hnwpef3T c_rally_46c61b3d_aYPvYjVl] Returning 500 to user: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.

  At several stages this error is found in the logs.

  Reuestid : req-25c9aee8-8713-4b6a-8f10-1b79a44c598c

  Openstack branch : stable/ocata
  Ubuntu controller version : 16.04.2
  Hypervisor : KVM
  stack@ocatacont:~/nsbu_cqe_openstack/devstack$ pip freeze | grep nova
  -e git+https://git.openstack.org//openstack/nova.git@506465a027dd1ba1c90949dc58297edae32da7e4#egg=nova
  python-novaclient==7.1.0

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1688220/+subscriptions
References

[Bug 1688220] [NEW] Operational error thrown by nova-api while serving the request
From: Ashish Kumar Gupta, 2017-05-04