yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1815082] [NEW] "DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Matt Riedemann <mriedem.os@xxxxxxxxx>
Date: Thu, 07 Feb 2019 16:15:48 -0000
Reply-to: Bug 1815082 <1815082@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Jens Harbott reported this in devstack:

https://review.openstack.org/#/c/635519/

He's running the n-api-meta service in the subnode per the multinode
guide:

https://docs.openstack.org/devstack/latest/guides/multinode-lab.html

However with current devstack, which doesn't configure nova.conf with
database access on the subnode, n-api-meta fails with this:

http://paste.openstack.org/show/744683/

Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: CRITICAL nova [None req-538e6b11-f91b-48d8-9c7a-d177dff7739b None None] Unhandled error: DBNonExistentTable: (sqlite3.OperationalError) no such t
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova Traceback (most recent call last):
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/bin/nova-metadata-wsgi", line 52, in <module>
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     application = init_application()
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/api/metadata/wsgi.py", line 20, in init_application
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return wsgi_app.init_application(NAME)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 82, in init_application
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     _setup_service(CONF.host, name)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 49, in _setup_service
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     ctxt, host, binary)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     result = fn(cls, context, *args, **kwargs)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/objects/service.py", line 334, in get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     host, binary)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/db/api.py", line 127, in service_get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return IMPL.service_get_by_host_and_binary(context, host, binary)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 242, in wrapped
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return f(context, *args, **kwargs)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 500, in service_get_by_host_and_binary
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     filter_by(binary=binary).\
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2979, in first
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     ret = list(self[0:1])
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return list(res)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return self._execute_and_instances(context)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     result = conn.execute(querycontext.statement, self._params)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 980, in execute
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return meth(self, multiparams, params)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return connection._execute_clauseelement(self, multiparams, params)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     distilled_params,
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     e, statement, parameters, cursor, context
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     util.raise_from_cause(newraise, exc_info)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     reraise(type(exception), exception, tb=exc_tb, cause=cause)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     cursor, statement, parameters, context
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     cursor.execute(statement, parameters)
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova DBNonExistentTable: (sqlite3.OperationalError) no such table: services [SQL: u'SELECT services.created_at AS services_created_at, serv
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova 
Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: unable to load app 0 (mountpoint='') (callable not found or import error)

I believe this is due to the nova-metadata service not configuring the
conductor indirection API when run in uwsgi mode like it is when run
with eventlet:

https://github.com/openstack/nova/blob/3cfcd117ce7ddb5a8d94396191cde559db73fb86/nova/cmd/api_metadata.py#L45

It also appears that we don't have any multinode CI jobs that run with
the n-api-meta service configured so we don't hit this in the gate.

The workaround which most deployments probably use is configuring
nova.conf with database access and run n-api-meta with that, even from
subnodes.

** Affects: nova
     Importance: Medium
         Status: New


** Tags: api metadata

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1815082

Title:
  "DBNonExistentTable: (sqlite3.OperationalError) no such table:
  services" when starting nova-metadata under uwsgi

Status in OpenStack Compute (nova):
  New

Bug description:
  Jens Harbott reported this in devstack:

  https://review.openstack.org/#/c/635519/

  He's running the n-api-meta service in the subnode per the multinode
  guide:

  https://docs.openstack.org/devstack/latest/guides/multinode-lab.html

  However with current devstack, which doesn't configure nova.conf with
  database access on the subnode, n-api-meta fails with this:

  http://paste.openstack.org/show/744683/

  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: CRITICAL nova [None req-538e6b11-f91b-48d8-9c7a-d177dff7739b None None] Unhandled error: DBNonExistentTable: (sqlite3.OperationalError) no such t
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova Traceback (most recent call last):
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/bin/nova-metadata-wsgi", line 52, in <module>
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     application = init_application()
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/api/metadata/wsgi.py", line 20, in init_application
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return wsgi_app.init_application(NAME)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 82, in init_application
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     _setup_service(CONF.host, name)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/api/openstack/wsgi_app.py", line 49, in _setup_service
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     ctxt, host, binary)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     result = fn(cls, context, *args, **kwargs)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/objects/service.py", line 334, in get_by_host_and_binary
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     host, binary)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/db/api.py", line 127, in service_get_by_host_and_binary
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return IMPL.service_get_by_host_and_binary(context, host, binary)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 242, in wrapped
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return f(context, *args, **kwargs)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 500, in service_get_by_host_and_binary
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     filter_by(binary=binary).\
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2979, in first
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     ret = list(self[0:1])
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2771, in __getitem__
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return list(res)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3081, in __iter__
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return self._execute_and_instances(context)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3106, in _execute_and_instances
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     result = conn.execute(querycontext.statement, self._params)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 980, in execute
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return meth(self, multiparams, params)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 273, in _execute_on_connection
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     return connection._execute_clauseelement(self, multiparams, params)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1099, in _execute_clauseelement
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     distilled_params,
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1240, in _execute_context
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     e, statement, parameters, cursor, context
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1456, in _handle_dbapi_exception
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     util.raise_from_cause(newraise, exc_info)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 296, in raise_from_cause
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     reraise(type(exception), exception, tb=exc_tb, cause=cause)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1236, in _execute_context
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     cursor, statement, parameters, context
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova   File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 536, in do_execute
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova     cursor.execute(statement, parameters)
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova DBNonExistentTable: (sqlite3.OperationalError) no such table: services [SQL: u'SELECT services.created_at AS services_created_at, serv
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: ERROR nova 
  Feb 07 13:33:32 jh-devstack01a devstack@n-api-meta.service[22651]: unable to load app 0 (mountpoint='') (callable not found or import error)

  I believe this is due to the nova-metadata service not configuring the
  conductor indirection API when run in uwsgi mode like it is when run
  with eventlet:

  https://github.com/openstack/nova/blob/3cfcd117ce7ddb5a8d94396191cde559db73fb86/nova/cmd/api_metadata.py#L45

  It also appears that we don't have any multinode CI jobs that run with
  the n-api-meta service configured so we don't hit this in the gate.

  The workaround which most deployments probably use is configuring
  nova.conf with database access and run n-api-meta with that, even from
  subnodes.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1815082/+subscriptions
Follow ups

[Bug 1815082] Re: "DBNonExistentTable: (sqlite3.OperationalError) no such table: services" when starting nova-metadata under uwsgi
From: Matt Riedemann, 2019-02-07