yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1642234] Re: error nova.api.openstack.extensions

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem@xxxxxxxxxx>
Date: Wed, 16 Nov 2016 16:38:27 -0000
Reply-to: Bug 1642234 <1642234@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
There is very little information in this bug report, like what release
of nova are you using? What's the env topology? There is obviously a
mysql connection failure, so you should probably investigate the mysql
deployment / config / logs as that's what nova is failing on:

2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions raise err.OperationalError(2006, "MySQL server has gone away (%r)" % (e,))
2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, 'Broken pipe'))")

** Changed in: nova
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1642234

Title:
  error nova.api.openstack.extensions

Status in OpenStack Compute (nova):
  Invalid

Bug description:
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters [req-3a504b21-c9c9-42e4-b246-9b506f47989b 33c709637a53464c9b459a80a64395c2 804207cd6c25496f92fa0f1d1c17d517 - - -] DBAPIError exception wrapped from (pymysql.err.InternalError) (1927, u'Connection was killed') [SQL: u'SELECT 1']
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     context)
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     cursor.execute(statement, parameters)
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 146, in execute
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     result = self._query(query)
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 296, in _query
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     conn.query(q)
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 781, in query
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 942, in _read_query_result
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     result.read()
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1138, in read
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     first_packet = self.connection._read_packet()
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 906, in _read_packet
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     packet.check_error()
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 367, in check_error
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     err.raise_mysql_exception(self._data)
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     _check_mysql_exception(errinfo)
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters     raise InternalError(errno, errorvalue)
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1927, u'Connection was killed')
  2016-11-16 19:44:18.942 2762 ERROR oslo_db.sqlalchemy.exc_filters
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions [req-3a504b21-c9c9-42e4-b246-9b506f47989b 33c709637a53464c9b459a80a64395c2 804207cd6c25496f92fa0f1d1c17d517 - - -] Unexpected exception in API method
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions Traceback (most recent call last):
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 478, in wrapped
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 194, in index
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     _services = self._get_services_list(req)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 87, in _get_services_list
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     _services = self._get_services(req)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/services.py", line 49, in _get_services
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     for s in self.host_api.service_get_all(context, set_zones=True)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3717, in service_get_all
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     set_zones=set_zones)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_versionedobjects/base.py", line 181, in wrapper
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     result = fn(cls, context, *args, **kwargs)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/objects/service.py", line 406, in get_all
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     db_services = db.service_get_all(context, disabled=disabled)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/api.py", line 136, in service_get_all
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     return IMPL.service_get_all(context, disabled)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 315, in wrapped
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     return f(context, *args, **kwargs)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/db/sqlalchemy/api.py", line 527, in service_get_all
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     return query.all()
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2588, in all
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     return list(self)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2736, in __iter__
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     return self._execute_and_instances(context)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in _execute_and_instances
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     close_with_result=True)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2740, in _connection_from_session
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     **kw)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 905, in connection
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     execution_options=execution_options)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     engine, execution_options)2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 910, in _connection_for_bind
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     engine, execution_options)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     conn = bind.contextual_connect()
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2041, in contextual_connect
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     **kwargs)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 92, in __init__
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     self.dispatch.engine_connect(self, self.__branch)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/event/attr.py", line 256, in __call__
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     fn(*args, **kw)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 72, in _connect_ping_listener
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     connection.scalar(select([1]))
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 844, in scalar
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     return self.execute(object, *multiparams, **params).scalar()
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in execute
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     return meth(self, multiparams, params)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in _execute_on_connection
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     return connection._execute_clauseelement(self, multiparams, params)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in _execute_clauseelement
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     compiled_sql, distilled_params
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     context)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1334, in _handle_dbapi_exception
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     self._autorollback()
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 791, in _autorollback
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     self._root._rollback_impl()
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 670, in _rollback_impl
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     self._handle_dbapi_exception(e, None, None, None, None)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1266, in _handle_dbapi_exception
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     exc_info
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 200, in raise_from_cause
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     reraise(type(exception), exception, tb=exc_tb)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 668, in _rollback_impl
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     self.engine.dialect.do_rollback(self.connection)
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2526, in do_rollback
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     dbapi_connection.rollback()
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 723, in rollback
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 970, in _execute_command
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     self._write_bytes(prelude + sql[:chunk_size-1])
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 929, in _write_bytes
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions     raise err.OperationalError(2006, "MySQL server has gone away (%r)" % (e,))
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(32, 'Broken pipe'))")
  2016-11-16 19:44:18.958 2762 ERROR nova.api.openstack.extensions
  2016-11-16 19:44:18.984 2762 INFO nova.api.openstack.wsgi [req-3a504b21-c9c9-42e4-b246-9b506f47989b 33c709637a53464c9b459a80a64395c2 804207cd6c25496f92fa0f1d1c17d517 - - -] HTTP exception thrown: Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible.
  <class 'sqlalchemy.exc.OperationalError'>
  2016-11-16 19:44:18.985 2762 INFO nova.osapi_compute.wsgi.server [req-3a504b21-c9c9-42e4-b246-9b506f47989b 33c709637a53464c9b459a80a64395c2 804207cd6c25496f92fa0f1d1c17d517 - - -] 192.168.153.130 "GET /v2.1/804207cd6c25496f92fa0f1d1c17d517/os-services HTTP/1.1" status: 500 len: 507 time: 0.7481852

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1642234/+subscriptions
References

[Bug 1642234] [NEW] error nova.api.openstack.extensions
From: rocky, 2016-11-16