yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1641021] Re: openstack create server cli genrates error

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Sat, 12 Nov 2016 00:10:18 -0000
Reply-to: Bug 1641021 <1641021@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
These are the errors in the API log:

2016-11-11 00:36:13.192 8718 ERROR oslo_service.service [req-f31b2dd1-6e1c-4aae-81ad-bc21f65255fd - - - - -] Error starting thread.
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service Traceback (most recent call last):
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_service/service.py", line 708, in run_service
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     service.start()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 357, in start
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     self.binary)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     result = fn(cls, context, *args, **kwargs)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/objects/service.py", line 252, in get_by_host_and_binary
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     host, binary)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 127, in service_get_by_host_and_binary
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return IMPL.service_get_by_host_and_binary(context, host, binary)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 254, in wrapped
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     with ctxt_mgr.reader.using(context):
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return self.gen.next()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 874, in _transaction_scope
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     allow_async=self._allow_async) as resource:
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return self.gen.next()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 518, in _session
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     bind=self.connection, mode=self.mode)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 277, in _create_session
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     self._start()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 363, in _start
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     engine_args, maker_args)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 387, in _setup_for_connection
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     sql_connection=sql_connection, **engine_kwargs)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 155, in create_engine
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     test_conn = _test_connection(engine, max_retries, retry_interval)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 331, in _test_connection
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return engine.connect()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2018, in connect
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return self._connection_cls(self, **kwargs)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 72, in __init__
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     if connection is not None else engine.raw_connection()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2104, in raw_connection
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     self.pool.unique_connection, _connection)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     e, dialect, self)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     util.raise_from_cause(newraise, exc_info)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     reraise(type(exception), exception, tb=exc_tb)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return fn()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 318, in unique_connection
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return _ConnectionFairy._checkout(self)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     fairy = _ConnectionRecord.checkout(pool)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     rec = pool._do_get()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     self._dec_overflow()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     compat.reraise(exc_type, exc_value, exc_tb)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return self._create_connection()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return _ConnectionRecord(self)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     self.connection = self.__connect()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     connection = self.__pool._invoke_creator(self)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return dialect.connect(*cargs, **cparams)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return self.dbapi.connect(*cargs, **cparams)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     return Connection(*args, **kwargs)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 679, in __init__
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     self.connect()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 891, in connect
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     self._request_authentication()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 1097, in _request_authentication
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     auth_packet = self._read_packet()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 966, in _read_packet
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     packet.check_error()
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pymysql/connections.py", line 394, in check_error
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     err.raise_mysql_exception(self._data)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 120, in raise_mysql_exception
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     _check_mysql_exception(errinfo)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service   File "/usr/lib/python2.7/dist-packages/pymysql/err.py", line 112, in _check_mysql_exception
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service     raise errorclass(errno, errorvalue)
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
2016-11-11 00:36:13.192 8718 ERROR oslo_service.service 

What size is the deployment? What is the MySQL configuration with
respect to number of open connections? What is the database
configuration in nova.conf (the oslo.db options)? This seems like a
deployment issue rather than a nova issue.

** Tags added: db

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1641021

Title:
  openstack create server cli genrates error

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  Openstack Newton (4 nodes. controller, neutron, nova, cinder)
  Ubuntu 16.04.1 LTS

  openstack server create --flavor m1.nano --image cirros   --nic net-id=048e2b61-dcd0-4af9-a8a9-f84e140cbea7  --security-group default   --key-name keysec selfservice-instance
  WARNING: openstackclient.common.utils is deprecated and will be removed after Jun 2017. Please use osc_lib.utils
  Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'> (HTTP 500) (Request-ID: req-8f3e873c-548a-4282-bee2-7e7ef8fc8419)

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1641021/+subscriptions
References

[Bug 1641021] [NEW] openstack create server cli genrates error
From: Frank-Charles Osafo, 2016-11-11