yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #76334
[Bug 1804271] Re: nova-api is broken in postgresql jobs
** Also affects: nova/rocky
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1804271
Title:
nova-api is broken in postgresql jobs
Status in OpenStack Compute (nova):
Fix Released
Status in OpenStack Compute (nova) rocky series:
In Progress
Bug description:
We noticed in neutron-tempest-postgres-full periodic job that since 14.11.2018 it's failing because of issue in nova-api.
Example of failure are on http://logs.openstack.org/periodic/git.openstack.org/openstack/neutron/master/neutron-tempest-postgres-full/1de7427/logs/screen-n-api.txt.gz?level=ERROR
Stacktrace:
Nov 20 06:34:20.578714 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi Traceback (most recent call last):
Nov 20 06:34:20.578885 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/openstack/wsgi.py", line 655, in wrapped
Nov 20 06:34:20.579046 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
Nov 20 06:34:20.579218 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.579379 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.579537 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.579737 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.579899 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.580070 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.580236 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.580396 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.580616 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.581036 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.581205 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.581364 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.581523 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.581693 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.581852 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.582008 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.582170 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.582336 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.582505 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.582663 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.582835 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/validation/__init__.py", line 110, in wrapper
Nov 20 06:34:20.582987 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
Nov 20 06:34:20.583285 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/api/openstack/compute/servers.py", line 596, in create
Nov 20 06:34:20.583453 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi **create_kwargs)
Nov 20 06:34:20.583651 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/hooks.py", line 154, in inner
Nov 20 06:34:20.583805 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi rv = f(*args, **kwargs)
Nov 20 06:34:20.583957 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/compute/api.py", line 1737, in create
Nov 20 06:34:20.584136 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi trusted_certs=trusted_certs)
Nov 20 06:34:20.584298 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/compute/api.py", line 1195, in _create_instance
Nov 20 06:34:20.584459 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi supports_multiattach, network_metadata)
Nov 20 06:34:20.584623 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/compute/api.py", line 869, in _provision_instances
Nov 20 06:34:20.585034 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi context, instance_type, min_count, max_count)
Nov 20 06:34:20.585199 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/compute/utils.py", line 967, in check_num_instances_quota
Nov 20 06:34:20.585358 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi check_user_id=user_id)
Nov 20 06:34:20.585530 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Nov 20 06:34:20.585708 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi result = fn(cls, context, *args, **kwargs)
Nov 20 06:34:20.585873 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/objects/quotas.py", line 339, in check_deltas
Nov 20 06:34:20.586037 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi **count_kwargs)
Nov 20 06:34:20.586195 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Nov 20 06:34:20.586363 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi result = fn(cls, context, *args, **kwargs)
Nov 20 06:34:20.586520 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/objects/quotas.py", line 302, in count_as_dict
Nov 20 06:34:20.586677 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi context, resource, *args, **kwargs)
Nov 20 06:34:20.586834 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/quota.py", line 1002, in count_as_dict
Nov 20 06:34:20.587002 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return res.count_as_dict(context, *args, **kwargs)
Nov 20 06:34:20.587160 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/quota.py", line 1178, in _instances_cores_ram_count
Nov 20 06:34:20.587324 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi context, project_id)
Nov 20 06:34:20.587481 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/objects/cell_mapping.py", line 292, in get_by_project_id
Nov 20 06:34:20.587721 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi db_mappings = cls._get_by_project_id_from_db(context, project_id)
Nov 20 06:34:20.587984 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1012, in wrapper
Nov 20 06:34:20.588247 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return fn(*args, **kwargs)
Nov 20 06:34:20.588493 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/opt/stack/new/nova/nova/objects/cell_mapping.py", line 283, in _get_by_project_id_from_db
Nov 20 06:34:20.588745 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi options(joinedload('cell_mapping', innerjoin=True)).\
Nov 20 06:34:20.588998 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2843, in all
Nov 20 06:34:20.589248 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return list(self)
Nov 20 06:34:20.589509 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2995, in __iter__
Nov 20 06:34:20.589783 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return self._execute_and_instances(context)
Nov 20 06:34:20.590043 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 3018, in _execute_and_instances
Nov 20 06:34:20.590316 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi result = conn.execute(querycontext.statement, self._params)
Nov 20 06:34:20.590639 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 948, in execute
Nov 20 06:34:20.590932 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return meth(self, multiparams, params)
Nov 20 06:34:20.591199 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
Nov 20 06:34:20.591459 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi return connection._execute_clauseelement(self, multiparams, params)
Nov 20 06:34:20.591746 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
Nov 20 06:34:20.592005 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi compiled_sql, distilled_params
Nov 20 06:34:20.592273 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
Nov 20 06:34:20.592540 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi context)
Nov 20 06:34:20.592806 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
Nov 20 06:34:20.593101 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi util.raise_from_cause(newraise, exc_info)
Nov 20 06:34:20.593558 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
Nov 20 06:34:20.593870 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi reraise(type(exception), exception, tb=exc_tb, cause=cause)
Nov 20 06:34:20.594130 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
Nov 20 06:34:20.594398 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi context)
Nov 20 06:34:20.594661 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 509, in do_execute
Nov 20 06:34:20.594937 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi cursor.execute(statement, parameters)
Nov 20 06:34:20.595226 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi DBError: (psycopg2.ProgrammingError) column "instance_mappings.created_at" must appear in the GROUP BY clause or be used in an aggregate function
Nov 20 06:34:20.595488 ubuntu-xenial-inap-mtl01-0000583998 devstack@n-api.service[24452]: ERROR nova.api.openstack.wsgi LINE 1: SELECT instance_mappings.created_at AS instance_mappings_cre...
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1804271/+subscriptions
References