yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2076430] Re: rally job broken with recent keystone change and fails with ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: yatin <2076430@xxxxxxxxxxxxxxxxxx>
Date: Tue, 13 Aug 2024 06:04:07 -0000
Reply-to: Bug 2076430 <2076430@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
<< Testing keystone revert in https://review.opendev.org/c/openstack/neutron/+/925968
The job passed with the revert.

<< Looks like the issue is with osprofiler being enabled for keystone in the neutron gate, but it's not being enabled in keystone gates, so that's why we didn't see the issue there.
@xek yes exactly that's why it got missed. Wouldn't be good to have some coverage in keystone gate as well for this?

<< So the fix should be to extend oslo_serialization to support date type and not to revert keystone patch.
Ok i can push a patch in oslo.serialization, but i also noticed the keystone patch is also being backported so oslo.serialization fixes would also require backports and releases and u-c update and that can take time so if this is the way would be good to hold keystone backports.

<< It is actually interesting that lots of other functional tests are not failing, that means the job is running some specific setup including osprofiler
@gtema it's not doing anything extra than enabling osprofiler, for test i did enabled it in devstack jobs and all those too failed[1][2] with same issue.

[1] https://review.opendev.org/c/openstack/devstack/+/926169/1
[2] https://zuul.opendev.org/t/openstack/buildset/b80b78109f344777afe00d02d62ef5fa


** Also affects: oslo.serialization
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/2076430

Title:
  rally job broken with recent keystone change and fails with
  ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive

Status in neutron:
  New
Status in oslo.serialization:
  In Progress

Bug description:
  Test fails as:-
  2024-08-07 17:06:05.650302 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients [-] Unable to authenticate for user c_rally_927546a8_h6aDLbnK in project c_rally_927546a8_ahUUmlp9: keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)
  2024-08-07 17:06:05.650941 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients Traceback (most recent call last):
  2024-08-07 17:06:05.651113 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients   File "/opt/stack/rally-openstack/rally_openstack/common/osclients.py", line 269, in auth_ref
  2024-08-07 17:06:05.651156 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients     self.cache["keystone_auth_ref"] = plugin.get_access(sess)
  2024-08-07 17:06:05.651193 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients   File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/base.py", line 131, in get_access
  2024-08-07 17:06:05.651229 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients     self.auth_ref = self.get_auth_ref(session)
  2024-08-07 17:06:05.651263 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients   File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/generic/base.py", line 205, in get_auth_ref
  2024-08-07 17:06:05.651334 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients     return self._plugin.get_auth_ref(session, **kwargs)
  2024-08-07 17:06:05.651348 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients   File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/base.py", line 185, in get_auth_ref
  2024-08-07 17:06:05.651356 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients     resp = session.post(token_url, json=body, headers=headers,
  2024-08-07 17:06:05.651363 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients   File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 1162, in post
  2024-08-07 17:06:05.651370 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients     return self.request(url, 'POST', **kwargs)
  2024-08-07 17:06:05.651377 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients   File "/opt/stack/data/venv/lib/python3.10/site-packages/keystoneauth1/session.py", line 985, in request
  2024-08-07 17:06:05.651384 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients     raise exceptions.from_response(resp, method, url)
  2024-08-07 17:06:05.651391 | controller | 2024-08-07 17:06:05.647 76504 ERROR rally_openstack.common.osclients keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)

  From keyston log:-
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: CRITICAL keystone [None req-96647e8e-2585-4279-80fa-c4fa97b8c455 None None] Unhandled error: ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone Traceback (most recent call last):
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1498, in __call__
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self.wsgi_app(environ, start_response)
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/proxy_fix.py", line 183, in __call__
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self.app(environ, start_response)
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     response = req.get_response(self.application)
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     status, headers, app_iter = self.call_application(
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 143, in __call__
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return resp(environ, start_response)
  Aug 07 17:03:06.408072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/base.py", line 124, in __call__
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     response = req.get_response(self.application)
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     status, headers, app_iter = self.call_application(
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/osprofiler/osprofiler/web.py", line 131, in __call__
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return request.get_response(self.application)
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     status, headers, app_iter = self.call_application(
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 07 17:03:06.409468 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_middleware/request_id.py", line 58, in __call__
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     response = req.get_response(self.application)
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     status, headers, app_iter = self.call_application(
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self.app(environ, start_response)
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 129, in __call__
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/dec.py", line 193, in call_func
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     response = req.get_response(self._app)
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1313, in send
  Aug 07 17:03:06.410473 np0038147114 devstack@keystone.service[56399]: ERROR keystone     status, headers, app_iter = self.call_application(
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/webob/request.py", line 1278, in call_application
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/werkzeug/middleware/dispatcher.py", line 81, in __call__
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return app(environ, start_response)
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1476, in wsgi_app
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone     response = self.handle_exception(e)
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return original_handler(e)
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return original_handler(e)
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return original_handler(e)
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   [Previous line repeated 28 more times]
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 1473, in wsgi_app
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone     response = self.full_dispatch_request()
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 882, in full_dispatch_request
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone     rv = self.handle_user_exception(e)
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return original_handler(e)
  Aug 07 17:03:06.411332 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return original_handler(e)
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 298, in error_router
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return original_handler(e)
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   [Previous line repeated 28 more times]
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 880, in full_dispatch_request
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     rv = self.dispatch_request()
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/app.py", line 865, in dispatch_request
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 489, in wrapper
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     resp = resource(*args, **kwargs)
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask/views.py", line 110, in view
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return current_app.ensure_sync(self.dispatch_request)(**kwargs)  # type: ignore[no-any-return]
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 604, in dispatch_request
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     resp = meth(*args, **kwargs)
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/common.py", line 1146, in wrapper
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return f(*args, **kwargs)
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/api/auth.py", line 328, in post
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     token = authentication.authenticate_for_token(auth_data)
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 202, in authenticate_for_token
  Aug 07 17:03:06.412219 np0038147114 devstack@keystone.service[56399]: ERROR keystone     authenticate(auth_info, auth_context)
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/api/_shared/authentication.py", line 170, in authenticate
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     resp = method.authenticate(auth_info.get_method_data(method_name))
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/auth/plugins/password.py", line 34, in authenticate
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     PROVIDERS.identity_api.authenticate(
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/common/manager.py", line 111, in wrapped
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     __ret_val = __f(*args, **kwargs)
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/notifications.py", line 718, in wrapper
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     result = f(wrapped_self, user_id, *args, **kwargs)
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 533, in wrapper
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return f(self, *args, **kwargs)
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 545, in wrapper
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return f(self, *args, **kwargs)
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 1069, in authenticate
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     PROVIDERS.shadow_users_api.set_last_active_at(ref['id'])
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/shadow_backends/sql.py", line 166, in set_last_active_at
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     with sql.session_for_write() as session:
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     next(self.gen)
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 56, in wrap_session
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     with sess as s:
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  Aug 07 17:03:06.413072 np0038147114 devstack@keystone.service[56399]: ERROR keystone     next(self.gen)
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 1042, in _transaction_scope
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     with current._produce_block(
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     next(self.gen)
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 649, in _session
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     with excutils.save_and_reraise_exception():
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     self.force_reraise()
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     raise self.value
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 647, in _session
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     self._end_session_transaction(self.session)
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 675, in _end_session_transaction
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     session.commit()
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 2017, in commit
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     trans.commit(_to_root=True)
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "<string>", line 2, in commit
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     ret_value = fn(self, *arg, **kw)
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1302, in commit
  Aug 07 17:03:06.414043 np0038147114 devstack@keystone.service[56399]: ERROR keystone     self._prepare_impl()
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "<string>", line 2, in _prepare_impl
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/state_changes.py", line 139, in _go
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone     ret_value = fn(self, *arg, **kw)
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 1277, in _prepare_impl
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone     self.session.flush()
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4341, in flush
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone     self._flush(objects)
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4476, in _flush
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone     with util.safe_reraise():
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone     raise exc_value.with_traceback(exc_tb)
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/session.py", line 4437, in _flush
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone     flush_context.execute()
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 466, in execute
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone     rec.execute(self)
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/unitofwork.py", line 642, in execute
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone     util.preloaded.orm_persistence.save_obj(
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 85, in save_obj
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone     _emit_update_statements(
  Aug 07 17:03:06.414927 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/orm/persistence.py", line 912, in _emit_update_statements
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     c = connection.execute(
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return meth(
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return connection._execute_clauseelement(
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1640, in _execute_clauseelement
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     ret = self._execute_context(
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return self._exec_single_context(
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1889, in _exec_single_context
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     str_statement, effective_parameters = fn(
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/sqlalchemy/engine/events.py", line 176, in wrap_before_cursor_execute
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     orig_fn(
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/osprofiler/osprofiler/sqlalchemy.py", line 71, in handler
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     profiler.start(name, info=info)
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/osprofiler/osprofiler/profiler.py", line 82, in start
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     profiler.start(name, info=info)
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/osprofiler/osprofiler/profiler.py", line 410, in start
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     self._notify("%s-start" % name, info)
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/osprofiler/osprofiler/profiler.py", line 436, in _notify
  Aug 07 17:03:06.415874 np0038147114 devstack@keystone.service[56399]: ERROR keystone     notifier.notify(payload)
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/osprofiler/osprofiler/notifier.py", line 38, in notify
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone     __notifier(info)
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/osprofiler/osprofiler/drivers/redis_driver.py", line 75, in notify
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone     self.db.lpush(key, jsonutils.dumps(data))
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 198, in dumps
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return json.dumps(obj, default=default, **kwargs)
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/usr/lib/python3.10/json/__init__.py", line 238, in dumps
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone     **kw).encode(obj)
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/usr/lib/python3.10/json/encoder.py", line 199, in encode
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone     chunks = self.iterencode(o, _one_shot=True)
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/usr/lib/python3.10/json/encoder.py", line 257, in iterencode
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone     return _iterencode(o, 0)
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone   File "/opt/stack/data/venv/lib/python3.10/site-packages/oslo_serialization/jsonutils.py", line 176, in to_primitive
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone     raise ValueError("Cannot convert %r to primitive" % (value,))
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: ERROR keystone
  Aug 07 17:03:06.416936 np0038147114 devstack@keystone.service[56399]: [pid: 56399|app: 0|req: 225/451] 200.225.47.15 () {70 vars in 1374 bytes} [Wed Aug  7 17:03:06 2024] POST /identity/v3/auth/tokens => generated 0 bytes in 61 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)

  Example failure:-
  https://2a378a06b97cbac17818-b80014e5a7a6453c822c4dc8f22159da.ssl.cf5.rackcdn.com/925376/2/check/neutron-
  ovs-rally-task/fb03014/job-output.txt

  also cinder-rally-tasks broken

  Builds:- https://zuul.opendev.org/t/openstack/builds?job_name=neutron-
  ovs-rally-task&job_name=cinder-rally-task&branch=master&skip=0

  Broken since patch
  https://review.opendev.org/c/openstack/keystone/+/924892

To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/2076430/+subscriptions
References

[Bug 2076430] [NEW] rally job broken with recent keystone change and fails with ValueError: Cannot convert datetime.date(2024, 8, 7) to primitive
From: yatin, 2024-08-09