yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #94406
[Bug 2076430] [NEW] rally job broken with recent keystone change and fails with ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
Public bug reported:
Test fails as:-
2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last):
2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref
2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.cache["keystone_auth_ref"] = plugin.get_access(sess)
2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access
2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.auth_ref = self.get_auth_ref(session)
2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref
2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self._plugin.get_auth_ref(session, **kwargs)
2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref
2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients resp = session.post(token_url, json=body, headers=headers,
2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post
2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self.request(url, 'POST', **kwargs)
2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request
2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients raise exceptions.from_response(resp, method, url)
2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
>From keyston log:-
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone Traceback (most recent call last):
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1498, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.wsgi_app(environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 143, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return resp(environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/web.py", line 131, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return request.get_response(self.application)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/request_id.py", line 58, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self._app)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return app(environ, start_response)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1476, in wsgi_app
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.handle_exception(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times]
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.full_dispatch_request()
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.handle_user_exception(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times]
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.dispatch_request()
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = resource(*args, **kwargs)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return]
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = meth(*args, **kwargs)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/common.py", line 1146, in wrapper
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(*args, **kwargs)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/auth.py", line 328, in post
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone token = authentication.authenticate_for_token(auth_data)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 202, in authenticate_for_token
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone authenticate(auth_info, auth_context)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 170, in authenticate
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = method.authenticate(auth_info.get_method_data(method_name))
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/auth/plugins/password.py", line 34, in authenticate
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.identity_api.authenticate(
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 111, in wrapped
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone __ret_val = __f(*args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/notifications.py", line 718, in wrapper
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone result = f(wrapped_self, user_id, *args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 533, in wrapper
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 545, in wrapper
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1069, in authenticate
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.shadow_users_api.set_last_active_at(ref['id'])
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/shadow_backends/sql.py", line 166, in set_last_active_at
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sql.session_for_write() as session:
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 56, in wrap_session
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sess as s:
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with current._produce_block(
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 649, in _session
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with excutils.save_and_reraise_exception():
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.force_reraise()
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise self.value
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 647, in _session
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._end_session_transaction(self.session)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 675, in _end_session_transaction
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone session.commit()
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone trans.commit(_to_root=True)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in commit
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._prepare_impl()
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in _prepare_impl
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.session.flush()
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._flush(objects)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4476, in _flush
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone with util.safe_reraise():
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise exc_value.with_traceback(exc_tb)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone flush_context.execute()
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone rec.execute(self)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone util.preloaded.orm_persistence.save_obj(
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone _emit_update_statements(
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone c = connection.execute(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return meth(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return connection._execute_clauseelement(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret = self._execute_context(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self._exec_single_context(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1889, in _exec_single_context
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone str_statement, effective_parameters = fn(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/events.py", line 176, in wrap_before_cursor_execute
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone orig_fn(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 71, in handler
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info)
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 82, in start
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info)
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 410, in start
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._notify("%s-start" % name, info)
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 436, in _notify
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone notifier.notify(payload)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/notifier.py", line 38, in notify
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone __notifier(info)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/drivers/redis_driver.py", line 75, in notify
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.db.lpush(key, jsonutils.dumps(data))
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 198, in dumps
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return json.dumps(obj, default=default, **kwargs)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone **kw).encode(obj)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone chunks = self.iterencode(o, _one_shot=True)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return _iterencode(o, 0)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 176, in to_primitive
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise ValueError("Cannot convert %r to primitive" % (value,))
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: [pid: 56399|app: 0|req: 225/451] 200.225.47.15 () {70 vars in 1374 bytes} [Wed Aug 7 17:03:06 2024] POST /identity/v3/auth/tokens => generated 0 bytes in 61 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
Example failure:-
https://2a378a06b97cbac17818-b80014e5a7a6453c822c4dc8f22159da.ssl.cf5.rackcdn.com/925376/2/check/neutron-
ovs-rally-task/fb03014/job-output.txt
also cinder-rally-tasks broken
Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-
ovs-rally-task&job_name=cinder-rally-task&branch=master&skip=0
Broken since patch
https://review.opendev.org/c/openstack/keystone/+/924892
** Affects: neutron
Importance: Undecided
Status: New
** Description changed:
Test fails as:-
2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last):
2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref
2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.cache["keystone_auth_ref"] = plugin.get_access(sess)
2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access
2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.auth_ref = self.get_auth_ref(session)
2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref
2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self._plugin.get_auth_ref(session, **kwargs)
2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref
2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients resp = session.post(token_url, json=body, headers=headers,
2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post
2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self.request(url, 'POST', **kwargs)
2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request
2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients raise exceptions.from_response(resp, method, url)
2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
From keyston log:-
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone Traceback (most recent call last):
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1498, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.wsgi_app(environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 143, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return resp(environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/web.py", line 131, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return request.get_response(self.application)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/request_id.py", line 58, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self._app)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return app(environ, start_response)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1476, in wsgi_app
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.handle_exception(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times]
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.full_dispatch_request()
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.handle_user_exception(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times]
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.dispatch_request()
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = resource(*args, **kwargs)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return]
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = meth(*args, **kwargs)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/common.py", line 1146, in wrapper
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(*args, **kwargs)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/auth.py", line 328, in post
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone token = authentication.authenticate_for_token(auth_data)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 202, in authenticate_for_token
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone authenticate(auth_info, auth_context)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 170, in authenticate
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = method.authenticate(auth_info.get_method_data(method_name))
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/auth/plugins/password.py", line 34, in authenticate
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.identity_api.authenticate(
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 111, in wrapped
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone __ret_val = __f(*args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/notifications.py", line 718, in wrapper
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone result = f(wrapped_self, user_id, *args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 533, in wrapper
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 545, in wrapper
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1069, in authenticate
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.shadow_users_api.set_last_active_at(ref['id'])
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/shadow_backends/sql.py", line 166, in set_last_active_at
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sql.session_for_write() as session:
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 56, in wrap_session
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sess as s:
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with current._produce_block(
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 649, in _session
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with excutils.save_and_reraise_exception():
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.force_reraise()
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise self.value
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 647, in _session
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._end_session_transaction(self.session)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 675, in _end_session_transaction
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone session.commit()
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone trans.commit(_to_root=True)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in commit
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._prepare_impl()
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in _prepare_impl
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.session.flush()
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._flush(objects)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4476, in _flush
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone with util.safe_reraise():
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise exc_value.with_traceback(exc_tb)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone flush_context.execute()
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone rec.execute(self)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone util.preloaded.orm_persistence.save_obj(
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone _emit_update_statements(
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone c = connection.execute(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return meth(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return connection._execute_clauseelement(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret = self._execute_context(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self._exec_single_context(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1889, in _exec_single_context
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone str_statement, effective_parameters = fn(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/events.py", line 176, in wrap_before_cursor_execute
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone orig_fn(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 71, in handler
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info)
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 82, in start
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info)
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 410, in start
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._notify("%s-start" % name, info)
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 436, in _notify
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone notifier.notify(payload)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/notifier.py", line 38, in notify
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone __notifier(info)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/drivers/redis_driver.py", line 75, in notify
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.db.lpush(key, jsonutils.dumps(data))
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 198, in dumps
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return json.dumps(obj, default=default, **kwargs)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone **kw).encode(obj)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone chunks = self.iterencode(o, _one_shot=True)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return _iterencode(o, 0)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 176, in to_primitive
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise ValueError("Cannot convert %r to primitive" % (value,))
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
- Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone
+ Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: [pid: 56399|app: 0|req: 225/451] 200.225.47.15 () {70 vars in 1374 bytes} [Wed Aug 7 17:03:06 2024] POST /identity/v3/auth/tokens => generated 0 bytes in 61 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
Example failure:-
https://2a378a06b97cbac17818-b80014e5a7a6453c822c4dc8f22159da.ssl.cf5.rackcdn.com/925376/2/check/neutron-
ovs-rally-task/fb03014/job-output.txt
also cinder-rally-tasks broken
Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-
ovs-rally-task&job_name=cinder-rally-task&branch=master&skip=0
+
+ Broken since patch
+ https://review.opendev.org/c/openstack/keystone/+/924892
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2076430
Title:
rally job broken with recent keystone change and fails with
ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
Status in neutron:
New
Bug description:
Test fails as:-
2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last):
2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref
2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.cache["keystone_auth_ref"] = plugin.get_access(sess)
2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access
2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients self.auth_ref = self.get_auth_ref(session)
2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref
2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self._plugin.get_auth_ref(session, **kwargs)
2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref
2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients resp = session.post(token_url, json=body, headers=headers,
2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post
2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients return self.request(url, 'POST', **kwargs)
2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request
2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients raise exceptions.from_response(resp, method, url)
2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
From keyston log:-
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone Traceback (most recent call last):
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1498, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.wsgi_app(environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 143, in __call__
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return resp(environ, start_response)
Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/web.py", line 131, in __call__
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone return request.get_response(self.application)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/request_id.py", line 58, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self.application)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.app(environ, start_response)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = req.get_response(self._app)
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone status, headers, app_iter = self.call_application(
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone app_iter = application(self.environ, start_response)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return app(environ, start_response)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1476, in wsgi_app
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.handle_exception(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times]
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone response = self.full_dispatch_request()
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.handle_user_exception(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return original_handler(e)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone [Previous line repeated 28 more times]
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone rv = self.dispatch_request()
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = resource(*args, **kwargs)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return current_app.ensure_sync(self.dispatch_request)(**kwargs) # type: ignore[no-any-return]
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = meth(*args, **kwargs)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/common.py", line 1146, in wrapper
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(*args, **kwargs)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/auth.py", line 328, in post
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone token = authentication.authenticate_for_token(auth_data)
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 202, in authenticate_for_token
Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone authenticate(auth_info, auth_context)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 170, in authenticate
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone resp = method.authenticate(auth_info.get_method_data(method_name))
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/auth/plugins/password.py", line 34, in authenticate
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.identity_api.authenticate(
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 111, in wrapped
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone __ret_val = __f(*args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/notifications.py", line 718, in wrapper
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone result = f(wrapped_self, user_id, *args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 533, in wrapper
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 545, in wrapper
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone return f(self, *args, **kwargs)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1069, in authenticate
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone PROVIDERS.shadow_users_api.set_last_active_at(ref['id'])
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/keystone/keystone/identity/shadow_backends/sql.py", line 166, in set_last_active_at
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sql.session_for_write() as session:
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen)
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 56, in wrap_session
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone with sess as s:
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with current._produce_block(
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone next(self.gen)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 649, in _session
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone with excutils.save_and_reraise_exception():
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.force_reraise()
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise self.value
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 647, in _session
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._end_session_transaction(self.session)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 675, in _end_session_transaction
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone session.commit()
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone trans.commit(_to_root=True)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in commit
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw)
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._prepare_impl()
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "<string>", line 2, in _prepare_impl
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret_value = fn(self, *arg, **kw)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.session.flush()
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._flush(objects)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4476, in _flush
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone with util.safe_reraise():
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise exc_value.with_traceback(exc_tb)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone flush_context.execute()
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone rec.execute(self)
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone util.preloaded.orm_persistence.save_obj(
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone _emit_update_statements(
Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone c = connection.execute(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return meth(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return connection._execute_clauseelement(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone ret = self._execute_context(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone return self._exec_single_context(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1889, in _exec_single_context
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone str_statement, effective_parameters = fn(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/events.py", line 176, in wrap_before_cursor_execute
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone orig_fn(
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 71, in handler
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info)
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 82, in start
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone profiler.start(name, info=info)
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 410, in start
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone self._notify("%s-start" % name, info)
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/profiler.py", line 436, in _notify
Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone notifier.notify(payload)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/notifier.py", line 38, in notify
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone __notifier(info)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/osprofiler/osprofiler/drivers/redis_driver.py", line 75, in notify
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone self.db.lpush(key, jsonutils.dumps(data))
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 198, in dumps
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return json.dumps(obj, default=default, **kwargs)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone **kw).encode(obj)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone chunks = self.iterencode(o, _one_shot=True)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone return _iterencode(o, 0)
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 176, in to_primitive
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone raise ValueError("Cannot convert %r to primitive" % (value,))
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone
Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: [pid: 56399|app: 0|req: 225/451] 200.225.47.15 () {70 vars in 1374 bytes} [Wed Aug 7 17:03:06 2024] POST /identity/v3/auth/tokens => generated 0 bytes in 61 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)
Example failure:-
https://2a378a06b97cbac17818-b80014e5a7a6453c822c4dc8f22159da.ssl.cf5.rackcdn.com/925376/2/check/neutron-
ovs-rally-task/fb03014/job-output.txt
also cinder-rally-tasks broken
Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-
ovs-rally-task&job_name=cinder-rally-task&branch=master&skip=0
Broken since patch
https://review.opendev.org/c/openstack/keystone/+/924892
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2076430/+subscriptions
Follow ups
