yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1926836] [NEW] Keystone Redis Caching

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Behzad <1926836@xxxxxxxxxxxxxxxxxx>
Date: Sat, 01 May 2021 12:27:16 -0000
Reply-to: Bug 1926836 <1926836@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

As trying to integrate the Keystone with Redis as the caching layer (the procedures are working fine with the Memcached), the following error is being shown:
The information:
OS = Ubuntu 20.04
Openstack repo: Ubuntu cloud (Wallaby edition)
keystone version 19.0.0

When i issue the "openstack endpoint list --service identity" command, and the output shows the "Internal Server Error (HTTP 500)".
====================================
The following error is the error shown in the /var/log/apache2/keystone.log:
here is the link for https://paste.ubuntu.com/p/CmrgZ2JrkX/
the following is the content of the link:

2021-05-01 16:37:17.350115 mod_wsgi (pid=241861): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'.
2021-05-01 16:37:17.358549 Traceback (most recent call last):
2021-05-01 16:37:17.359016   File "/usr/lib/python3/dist-packages/flask/app.py", line 2463, in __call__
2021-05-01 16:37:17.359044     return self.wsgi_app(environ, start_response)
2021-05-01 16:37:17.359072   File "/usr/lib/python3/dist-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
2021-05-01 16:37:17.359083     return self.app(environ, start_response)
2021-05-01 16:37:17.359104   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2021-05-01 16:37:17.359114     resp = self.call_func(req, *args, **kw)
2021-05-01 16:37:17.359134   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2021-05-01 16:37:17.359144     return self.func(req, *args, **kwargs)
2021-05-01 16:37:17.359165   File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 124, in __call__
2021-05-01 16:37:17.359174     response = req.get_response(self.application)
2021-05-01 16:37:17.359195   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
2021-05-01 16:37:17.359204     status, headers, app_iter = self.call_application(
2021-05-01 16:37:17.359225   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2021-05-01 16:37:17.359239     app_iter = application(self.environ, start_response)
2021-05-01 16:37:17.359260   File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__
2021-05-01 16:37:17.359271     return resp(environ, start_response)
2021-05-01 16:37:17.359293   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2021-05-01 16:37:17.359304     resp = self.call_func(req, *args, **kw)
2021-05-01 16:37:17.359325   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2021-05-01 16:37:17.359339     return self.func(req, *args, **kwargs)
2021-05-01 16:37:17.359363   File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 124, in __call__
2021-05-01 16:37:17.359373     response = req.get_response(self.application)
2021-05-01 16:37:17.359395   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
2021-05-01 16:37:17.359406     status, headers, app_iter = self.call_application(
2021-05-01 16:37:17.359429   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2021-05-01 16:37:17.359439     app_iter = application(self.environ, start_response)
2021-05-01 16:37:17.359460   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2021-05-01 16:37:17.359470     resp = self.call_func(req, *args, **kw)
2021-05-01 16:37:17.359492   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2021-05-01 16:37:17.359502     return self.func(req, *args, **kwargs)
2021-05-01 16:37:17.359523   File "/usr/lib/python3/dist-packages/osprofiler/web.py", line 112, in __call__
2021-05-01 16:37:17.359534     return request.get_response(self.application)
2021-05-01 16:37:17.359579   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
2021-05-01 16:37:17.359589     status, headers, app_iter = self.call_application(
2021-05-01 16:37:17.359611   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2021-05-01 16:37:17.359625     app_iter = application(self.environ, start_response)
2021-05-01 16:37:17.359648   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2021-05-01 16:37:17.359658     resp = self.call_func(req, *args, **kw)
2021-05-01 16:37:17.359679   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2021-05-01 16:37:17.359688     return self.func(req, *args, **kwargs)
2021-05-01 16:37:17.359709   File "/usr/lib/python3/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
2021-05-01 16:37:17.359722     response = req.get_response(self.application)
2021-05-01 16:37:17.359745   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
2021-05-01 16:37:17.359754     status, headers, app_iter = self.call_application(
2021-05-01 16:37:17.359776   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2021-05-01 16:37:17.359785     app_iter = application(self.environ, start_response)
2021-05-01 16:37:17.359806   File "/usr/lib/python3/dist-packages/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
2021-05-01 16:37:17.359817     return self.app(environ, start_response)
2021-05-01 16:37:17.359839   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2021-05-01 16:37:17.359849     resp = self.call_func(req, *args, **kw)
2021-05-01 16:37:17.359871   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2021-05-01 16:37:17.359880     return self.func(req, *args, **kwargs)
2021-05-01 16:37:17.359902   File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
2021-05-01 16:37:17.359912     response = req.get_response(self._app)
2021-05-01 16:37:17.359933   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
2021-05-01 16:37:17.359942     status, headers, app_iter = self.call_application(
2021-05-01 16:37:17.359963   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2021-05-01 16:37:17.359973     app_iter = application(self.environ, start_response)
2021-05-01 16:37:17.359994   File "/usr/lib/python3/dist-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__
2021-05-01 16:37:17.360004     return app(environ, start_response)
2021-05-01 16:37:17.360026   File "/usr/lib/python3/dist-packages/flask/app.py", line 2449, in wsgi_app
2021-05-01 16:37:17.360036     response = self.handle_exception(e)
2021-05-01 16:37:17.360057   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
2021-05-01 16:37:17.360066     return original_handler(e)
2021-05-01 16:37:17.360087   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
2021-05-01 16:37:17.360097     return original_handler(e)
2021-05-01 16:37:17.360118   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
2021-05-01 16:37:17.360127     return original_handler(e)
2021-05-01 16:37:17.360148   [Previous line repeated 27 more times]
2021-05-01 16:37:17.360171   File "/usr/lib/python3/dist-packages/flask/app.py", line 1866, in handle_exception
2021-05-01 16:37:17.360182     reraise(exc_type, exc_value, tb)
2021-05-01 16:37:17.360203   File "/usr/lib/python3/dist-packages/flask/_compat.py", line 38, in reraise
2021-05-01 16:37:17.360213     raise value.with_traceback(tb)
2021-05-01 16:37:17.360233   File "/usr/lib/python3/dist-packages/flask/app.py", line 2446, in wsgi_app
2021-05-01 16:37:17.360243     response = self.full_dispatch_request()
2021-05-01 16:37:17.360264   File "/usr/lib/python3/dist-packages/flask/app.py", line 1951, in full_dispatch_request
2021-05-01 16:37:17.360284     rv = self.handle_user_exception(e)
2021-05-01 16:37:17.360308   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
2021-05-01 16:37:17.360317     return original_handler(e)
2021-05-01 16:37:17.360338   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
2021-05-01 16:37:17.360352     return original_handler(e)
2021-05-01 16:37:17.360375   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
2021-05-01 16:37:17.360385     return original_handler(e)
2021-05-01 16:37:17.360406   [Previous line repeated 27 more times]
2021-05-01 16:37:17.360429   File "/usr/lib/python3/dist-packages/flask/app.py", line 1820, in handle_user_exception
2021-05-01 16:37:17.360440     reraise(exc_type, exc_value, tb)
2021-05-01 16:37:17.360461   File "/usr/lib/python3/dist-packages/flask/_compat.py", line 38, in reraise
2021-05-01 16:37:17.360471     raise value.with_traceback(tb)
2021-05-01 16:37:17.360493   File "/usr/lib/python3/dist-packages/flask/app.py", line 1949, in full_dispatch_request
2021-05-01 16:37:17.360502     rv = self.dispatch_request()
2021-05-01 16:37:17.360523   File "/usr/lib/python3/dist-packages/flask/app.py", line 1935, in dispatch_request
2021-05-01 16:37:17.360533     return self.view_functions[rule.endpoint](**req.view_args)
2021-05-01 16:37:17.360554   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 468, in wrapper
2021-05-01 16:37:17.360565     resp = resource(*args, **kwargs)
2021-05-01 16:37:17.360586   File "/usr/lib/python3/dist-packages/flask/views.py", line 89, in view
2021-05-01 16:37:17.360596     return self.dispatch_request(*args, **kwargs)
2021-05-01 16:37:17.360617   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 583, in dispatch_request
2021-05-01 16:37:17.360627     resp = meth(*args, **kwargs)
2021-05-01 16:37:17.360648   File "/usr/lib/python3/dist-packages/keystone/server/flask/common.py", line 1064, in wrapper
2021-05-01 16:37:17.360659     return f(*args, **kwargs)
2021-05-01 16:37:17.360680   File "/usr/lib/python3/dist-packages/keystone/api/auth.py", line 315, in post
2021-05-01 16:37:17.360690     token = authentication.authenticate_for_token(auth_data)
2021-05-01 16:37:17.360711   File "/usr/lib/python3/dist-packages/keystone/api/_shared/authentication.py", line 182, in authenticate_for_token
2021-05-01 16:37:17.360721     auth_info = core.AuthInfo.create(auth=auth)
2021-05-01 16:37:17.360743   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 126, in create
2021-05-01 16:37:17.360752     auth_info._validate_and_normalize_auth_data(scope_only)
2021-05-01 16:37:17.360773   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 340, in _validate_and_normalize_auth_data
2021-05-01 16:37:17.360787     self._validate_and_normalize_scope_data()
2021-05-01 16:37:17.360810   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 290, in _validate_and_normalize_scope_data
2021-05-01 16:37:17.360820     project_ref = self._lookup_project(self.auth['scope']['project'])
2021-05-01 16:37:17.360841   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 196, in _lookup_project
2021-05-01 16:37:17.360851     domain_ref = self._lookup_domain(project_info['domain'])
2021-05-01 16:37:17.360873   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 171, in _lookup_domain
2021-05-01 16:37:17.360882     domain_ref = PROVIDERS.resource_api.get_domain_by_name(
2021-05-01 16:37:17.360903   File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 115, in wrapped
2021-05-01 16:37:17.360914     __ret_val = __f(*args, **kwargs)
2021-05-01 16:37:17.360935   File "<decorator-gen-28>", line 2, in get_domain_by_name
2021-05-01 16:37:17.360958   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
2021-05-01 16:37:17.360969     return self.get_or_create(
2021-05-01 16:37:17.360990   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 974, in get_or_create
2021-05-01 16:37:17.361055     key = self.key_mangler(key)
2021-05-01 16:37:17.361083   File "/usr/lib/python3/dist-packages/keystone/common/cache/core.py", line 86, in key_mangler
2021-05-01 16:37:17.361095     key = '%s:%s' % (key, invalidation_manager.region_id)
2021-05-01 16:37:17.361117   File "/usr/lib/python3/dist-packages/keystone/common/cache/core.py", line 43, in region_id
2021-05-01 16:37:17.361126     return self._invalidation_region.get_or_create(
2021-05-01 16:37:17.361148   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create
2021-05-01 16:37:17.361158     with Lock(
2021-05-01 16:37:17.361179   File "/usr/lib/python3/dist-packages/dogpile/lock.py", line 185, in __enter__
2021-05-01 16:37:17.361189     return self._enter()
2021-05-01 16:37:17.361209   File "/usr/lib/python3/dist-packages/dogpile/lock.py", line 87, in _enter
2021-05-01 16:37:17.361223     value = value_fn()
2021-05-01 16:37:17.361246   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 977, in get_value
2021-05-01 16:37:17.361256     value = self._get_from_backend(key)
2021-05-01 16:37:17.361277   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1265, in _get_from_backend
2021-05-01 16:37:17.361286     self.backend.get_serialized(key)
2021-05-01 16:37:17.361307   File "/usr/lib/python3/dist-packages/dogpile/cache/api.py", line 288, in get_serialized
2021-05-01 16:37:17.361317     return cast(SerializedReturnType, self.get(key))
2021-05-01 16:37:17.361338   File "/usr/lib/python3/dist-packages/keystone/common/cache/_context_cache.py", line 74, in get
2021-05-01 16:37:17.361348     value = self.proxied.get(key)
2021-05-01 16:37:17.361369   File "/usr/lib/python3/dist-packages/dogpile/cache/api.py", line 245, in get
2021-05-01 16:37:17.361378     raise NotImplementedError()
2021-05-01 16:37:17.361460 NotImplementedError

follwoing is the keystone.conf configuration for both Apache:
========================================================
cat /etc/apache2/sites-enabled/keystone.conf 
Listen 5000

<VirtualHost *:5000>
    WSGIScriptAlias / /usr/bin/keystone-wsgi-public
    WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
    WSGIProcessGroup keystone-public
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
    LimitRequestBody 114688

    <IfVersion >= 2.4>
      ErrorLogFormat "%{cu}t %M"
    </IfVersion>

    ErrorLog /var/log/apache2/keystone.log
    CustomLog /var/log/apache2/keystone_access.log combined

    <Directory /usr/bin>
        <IfVersion >= 2.4>
                Require all granted
        </IfVersion>
        <IfVersion < 2.4>
            Order allow,deny
            Allow from all
        </IfVersion>
    </Directory>
</VirtualHost>

Alias /identity /usr/bin/keystone-wsgi-public
<Location /identity>
    SetHandler wsgi-script
    Options +ExecCGI

    WSGIProcessGroup keystone-public
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On
</Location>

====================================
Content of /etc/keystone/keystone.conf

[DEFAULT]
log_dir = /var/log/keystone
default_log_levels = oslo.cache=DEBUG,dogpile.core.dogpile=DEBUG
[application_credential]
[assignment]
[auth]
[cache]
debug_cache_backend = True
backend = dogpile.cache.redis
backend_argument=url:redis://10.10.13.210/6379/0
enabled = true
[catalog]
[cors]
[credential]
[database]
connection = mysql+pymysql://keystone:PointofView@xxxxxxxxxxxxxxxxxxxx/keystone
[domain_config]
[endpoint_filter]
[endpoint_policy]
[eventlet_server]
[extra_headers]
Distribution = Ubuntu
[federation]
[fernet_receipts]
[fernet_tokens]
[healthcheck]
[identity]
[identity_mapping]
[jwt_tokens]
[ldap]
[memcache]
[oauth1]
[oslo_messaging_amqp]
[oslo_messaging_kafka]
[oslo_messaging_notifications]
[oslo_messaging_rabbit]
[oslo_middleware]
[oslo_policy]
[policy]
[profiler]
[receipt]
[resource]
[revoke]
[role]
[saml]
[security_compliance]
[shadow_users]
[token]
provider = fernet
caching = true
driver = keystone.token.persistence.backends.kvs.Token
[tokenless_auth]
[totp]
[trust]
[unified_limit]
[wsgi]

** Affects: keystone
     Importance: Undecided
         Status: New


** Tags: redis

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1926836

Title:
  Keystone Redis Caching

Status in OpenStack Identity (keystone):
  New

Bug description:
  As trying to integrate the Keystone with Redis as the caching layer (the procedures are working fine with the Memcached), the following error is being shown:
  The information:
  OS = Ubuntu 20.04
  Openstack repo: Ubuntu cloud (Wallaby edition)
  keystone version 19.0.0

  When i issue the "openstack endpoint list --service identity" command, and the output shows the "Internal Server Error (HTTP 500)".
  ====================================
  The following error is the error shown in the /var/log/apache2/keystone.log:
  here is the link for https://paste.ubuntu.com/p/CmrgZ2JrkX/
  the following is the content of the link:

  2021-05-01 16:37:17.350115 mod_wsgi (pid=241861): Exception occurred processing WSGI script '/usr/bin/keystone-wsgi-public'.
  2021-05-01 16:37:17.358549 Traceback (most recent call last):
  2021-05-01 16:37:17.359016   File "/usr/lib/python3/dist-packages/flask/app.py", line 2463, in __call__
  2021-05-01 16:37:17.359044     return self.wsgi_app(environ, start_response)
  2021-05-01 16:37:17.359072   File "/usr/lib/python3/dist-packages/werkzeug/middleware/proxy_fix.py", line 232, in __call__
  2021-05-01 16:37:17.359083     return self.app(environ, start_response)
  2021-05-01 16:37:17.359104   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
  2021-05-01 16:37:17.359114     resp = self.call_func(req, *args, **kw)
  2021-05-01 16:37:17.359134   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
  2021-05-01 16:37:17.359144     return self.func(req, *args, **kwargs)
  2021-05-01 16:37:17.359165   File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 124, in __call__
  2021-05-01 16:37:17.359174     response = req.get_response(self.application)
  2021-05-01 16:37:17.359195   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
  2021-05-01 16:37:17.359204     status, headers, app_iter = self.call_application(
  2021-05-01 16:37:17.359225   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
  2021-05-01 16:37:17.359239     app_iter = application(self.environ, start_response)
  2021-05-01 16:37:17.359260   File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__
  2021-05-01 16:37:17.359271     return resp(environ, start_response)
  2021-05-01 16:37:17.359293   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
  2021-05-01 16:37:17.359304     resp = self.call_func(req, *args, **kw)
  2021-05-01 16:37:17.359325   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
  2021-05-01 16:37:17.359339     return self.func(req, *args, **kwargs)
  2021-05-01 16:37:17.359363   File "/usr/lib/python3/dist-packages/oslo_middleware/base.py", line 124, in __call__
  2021-05-01 16:37:17.359373     response = req.get_response(self.application)
  2021-05-01 16:37:17.359395   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
  2021-05-01 16:37:17.359406     status, headers, app_iter = self.call_application(
  2021-05-01 16:37:17.359429   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
  2021-05-01 16:37:17.359439     app_iter = application(self.environ, start_response)
  2021-05-01 16:37:17.359460   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
  2021-05-01 16:37:17.359470     resp = self.call_func(req, *args, **kw)
  2021-05-01 16:37:17.359492   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
  2021-05-01 16:37:17.359502     return self.func(req, *args, **kwargs)
  2021-05-01 16:37:17.359523   File "/usr/lib/python3/dist-packages/osprofiler/web.py", line 112, in __call__
  2021-05-01 16:37:17.359534     return request.get_response(self.application)
  2021-05-01 16:37:17.359579   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
  2021-05-01 16:37:17.359589     status, headers, app_iter = self.call_application(
  2021-05-01 16:37:17.359611   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
  2021-05-01 16:37:17.359625     app_iter = application(self.environ, start_response)
  2021-05-01 16:37:17.359648   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
  2021-05-01 16:37:17.359658     resp = self.call_func(req, *args, **kw)
  2021-05-01 16:37:17.359679   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
  2021-05-01 16:37:17.359688     return self.func(req, *args, **kwargs)
  2021-05-01 16:37:17.359709   File "/usr/lib/python3/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
  2021-05-01 16:37:17.359722     response = req.get_response(self.application)
  2021-05-01 16:37:17.359745   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
  2021-05-01 16:37:17.359754     status, headers, app_iter = self.call_application(
  2021-05-01 16:37:17.359776   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
  2021-05-01 16:37:17.359785     app_iter = application(self.environ, start_response)
  2021-05-01 16:37:17.359806   File "/usr/lib/python3/dist-packages/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
  2021-05-01 16:37:17.359817     return self.app(environ, start_response)
  2021-05-01 16:37:17.359839   File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
  2021-05-01 16:37:17.359849     resp = self.call_func(req, *args, **kw)
  2021-05-01 16:37:17.359871   File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
  2021-05-01 16:37:17.359880     return self.func(req, *args, **kwargs)
  2021-05-01 16:37:17.359902   File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
  2021-05-01 16:37:17.359912     response = req.get_response(self._app)
  2021-05-01 16:37:17.359933   File "/usr/lib/python3/dist-packages/webob/request.py", line 1313, in send
  2021-05-01 16:37:17.359942     status, headers, app_iter = self.call_application(
  2021-05-01 16:37:17.359963   File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
  2021-05-01 16:37:17.359973     app_iter = application(self.environ, start_response)
  2021-05-01 16:37:17.359994   File "/usr/lib/python3/dist-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__
  2021-05-01 16:37:17.360004     return app(environ, start_response)
  2021-05-01 16:37:17.360026   File "/usr/lib/python3/dist-packages/flask/app.py", line 2449, in wsgi_app
  2021-05-01 16:37:17.360036     response = self.handle_exception(e)
  2021-05-01 16:37:17.360057   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
  2021-05-01 16:37:17.360066     return original_handler(e)
  2021-05-01 16:37:17.360087   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
  2021-05-01 16:37:17.360097     return original_handler(e)
  2021-05-01 16:37:17.360118   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
  2021-05-01 16:37:17.360127     return original_handler(e)
  2021-05-01 16:37:17.360148   [Previous line repeated 27 more times]
  2021-05-01 16:37:17.360171   File "/usr/lib/python3/dist-packages/flask/app.py", line 1866, in handle_exception
  2021-05-01 16:37:17.360182     reraise(exc_type, exc_value, tb)
  2021-05-01 16:37:17.360203   File "/usr/lib/python3/dist-packages/flask/_compat.py", line 38, in reraise
  2021-05-01 16:37:17.360213     raise value.with_traceback(tb)
  2021-05-01 16:37:17.360233   File "/usr/lib/python3/dist-packages/flask/app.py", line 2446, in wsgi_app
  2021-05-01 16:37:17.360243     response = self.full_dispatch_request()
  2021-05-01 16:37:17.360264   File "/usr/lib/python3/dist-packages/flask/app.py", line 1951, in full_dispatch_request
  2021-05-01 16:37:17.360284     rv = self.handle_user_exception(e)
  2021-05-01 16:37:17.360308   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
  2021-05-01 16:37:17.360317     return original_handler(e)
  2021-05-01 16:37:17.360338   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
  2021-05-01 16:37:17.360352     return original_handler(e)
  2021-05-01 16:37:17.360375   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 272, in error_router
  2021-05-01 16:37:17.360385     return original_handler(e)
  2021-05-01 16:37:17.360406   [Previous line repeated 27 more times]
  2021-05-01 16:37:17.360429   File "/usr/lib/python3/dist-packages/flask/app.py", line 1820, in handle_user_exception
  2021-05-01 16:37:17.360440     reraise(exc_type, exc_value, tb)
  2021-05-01 16:37:17.360461   File "/usr/lib/python3/dist-packages/flask/_compat.py", line 38, in reraise
  2021-05-01 16:37:17.360471     raise value.with_traceback(tb)
  2021-05-01 16:37:17.360493   File "/usr/lib/python3/dist-packages/flask/app.py", line 1949, in full_dispatch_request
  2021-05-01 16:37:17.360502     rv = self.dispatch_request()
  2021-05-01 16:37:17.360523   File "/usr/lib/python3/dist-packages/flask/app.py", line 1935, in dispatch_request
  2021-05-01 16:37:17.360533     return self.view_functions[rule.endpoint](**req.view_args)
  2021-05-01 16:37:17.360554   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 468, in wrapper
  2021-05-01 16:37:17.360565     resp = resource(*args, **kwargs)
  2021-05-01 16:37:17.360586   File "/usr/lib/python3/dist-packages/flask/views.py", line 89, in view
  2021-05-01 16:37:17.360596     return self.dispatch_request(*args, **kwargs)
  2021-05-01 16:37:17.360617   File "/usr/lib/python3/dist-packages/flask_restful/__init__.py", line 583, in dispatch_request
  2021-05-01 16:37:17.360627     resp = meth(*args, **kwargs)
  2021-05-01 16:37:17.360648   File "/usr/lib/python3/dist-packages/keystone/server/flask/common.py", line 1064, in wrapper
  2021-05-01 16:37:17.360659     return f(*args, **kwargs)
  2021-05-01 16:37:17.360680   File "/usr/lib/python3/dist-packages/keystone/api/auth.py", line 315, in post
  2021-05-01 16:37:17.360690     token = authentication.authenticate_for_token(auth_data)
  2021-05-01 16:37:17.360711   File "/usr/lib/python3/dist-packages/keystone/api/_shared/authentication.py", line 182, in authenticate_for_token
  2021-05-01 16:37:17.360721     auth_info = core.AuthInfo.create(auth=auth)
  2021-05-01 16:37:17.360743   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 126, in create
  2021-05-01 16:37:17.360752     auth_info._validate_and_normalize_auth_data(scope_only)
  2021-05-01 16:37:17.360773   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 340, in _validate_and_normalize_auth_data
  2021-05-01 16:37:17.360787     self._validate_and_normalize_scope_data()
  2021-05-01 16:37:17.360810   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 290, in _validate_and_normalize_scope_data
  2021-05-01 16:37:17.360820     project_ref = self._lookup_project(self.auth['scope']['project'])
  2021-05-01 16:37:17.360841   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 196, in _lookup_project
  2021-05-01 16:37:17.360851     domain_ref = self._lookup_domain(project_info['domain'])
  2021-05-01 16:37:17.360873   File "/usr/lib/python3/dist-packages/keystone/auth/core.py", line 171, in _lookup_domain
  2021-05-01 16:37:17.360882     domain_ref = PROVIDERS.resource_api.get_domain_by_name(
  2021-05-01 16:37:17.360903   File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 115, in wrapped
  2021-05-01 16:37:17.360914     __ret_val = __f(*args, **kwargs)
  2021-05-01 16:37:17.360935   File "<decorator-gen-28>", line 2, in get_domain_by_name
  2021-05-01 16:37:17.360958   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
  2021-05-01 16:37:17.360969     return self.get_or_create(
  2021-05-01 16:37:17.360990   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 974, in get_or_create
  2021-05-01 16:37:17.361055     key = self.key_mangler(key)
  2021-05-01 16:37:17.361083   File "/usr/lib/python3/dist-packages/keystone/common/cache/core.py", line 86, in key_mangler
  2021-05-01 16:37:17.361095     key = '%s:%s' % (key, invalidation_manager.region_id)
  2021-05-01 16:37:17.361117   File "/usr/lib/python3/dist-packages/keystone/common/cache/core.py", line 43, in region_id
  2021-05-01 16:37:17.361126     return self._invalidation_region.get_or_create(
  2021-05-01 16:37:17.361148   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create
  2021-05-01 16:37:17.361158     with Lock(
  2021-05-01 16:37:17.361179   File "/usr/lib/python3/dist-packages/dogpile/lock.py", line 185, in __enter__
  2021-05-01 16:37:17.361189     return self._enter()
  2021-05-01 16:37:17.361209   File "/usr/lib/python3/dist-packages/dogpile/lock.py", line 87, in _enter
  2021-05-01 16:37:17.361223     value = value_fn()
  2021-05-01 16:37:17.361246   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 977, in get_value
  2021-05-01 16:37:17.361256     value = self._get_from_backend(key)
  2021-05-01 16:37:17.361277   File "/usr/lib/python3/dist-packages/dogpile/cache/region.py", line 1265, in _get_from_backend
  2021-05-01 16:37:17.361286     self.backend.get_serialized(key)
  2021-05-01 16:37:17.361307   File "/usr/lib/python3/dist-packages/dogpile/cache/api.py", line 288, in get_serialized
  2021-05-01 16:37:17.361317     return cast(SerializedReturnType, self.get(key))
  2021-05-01 16:37:17.361338   File "/usr/lib/python3/dist-packages/keystone/common/cache/_context_cache.py", line 74, in get
  2021-05-01 16:37:17.361348     value = self.proxied.get(key)
  2021-05-01 16:37:17.361369   File "/usr/lib/python3/dist-packages/dogpile/cache/api.py", line 245, in get
  2021-05-01 16:37:17.361378     raise NotImplementedError()
  2021-05-01 16:37:17.361460 NotImplementedError

  follwoing is the keystone.conf configuration for both Apache:
  ========================================================
  cat /etc/apache2/sites-enabled/keystone.conf 
  Listen 5000

  <VirtualHost *:5000>
      WSGIScriptAlias / /usr/bin/keystone-wsgi-public
      WSGIDaemonProcess keystone-public processes=5 threads=1 user=keystone group=keystone display-name=%{GROUP}
      WSGIProcessGroup keystone-public
      WSGIApplicationGroup %{GLOBAL}
      WSGIPassAuthorization On
      LimitRequestBody 114688

      <IfVersion >= 2.4>
        ErrorLogFormat "%{cu}t %M"
      </IfVersion>

      ErrorLog /var/log/apache2/keystone.log
      CustomLog /var/log/apache2/keystone_access.log combined

      <Directory /usr/bin>
          <IfVersion >= 2.4>
                  Require all granted
          </IfVersion>
          <IfVersion < 2.4>
              Order allow,deny
              Allow from all
          </IfVersion>
      </Directory>
  </VirtualHost>

  Alias /identity /usr/bin/keystone-wsgi-public
  <Location /identity>
      SetHandler wsgi-script
      Options +ExecCGI

      WSGIProcessGroup keystone-public
      WSGIApplicationGroup %{GLOBAL}
      WSGIPassAuthorization On
  </Location>

  ====================================
  Content of /etc/keystone/keystone.conf

  [DEFAULT]
  log_dir = /var/log/keystone
  default_log_levels = oslo.cache=DEBUG,dogpile.core.dogpile=DEBUG
  [application_credential]
  [assignment]
  [auth]
  [cache]
  debug_cache_backend = True
  backend = dogpile.cache.redis
  backend_argument=url:redis://10.10.13.210/6379/0
  enabled = true
  [catalog]
  [cors]
  [credential]
  [database]
  connection = mysql+pymysql://keystone:PointofView@xxxxxxxxxxxxxxxxxxxx/keystone
  [domain_config]
  [endpoint_filter]
  [endpoint_policy]
  [eventlet_server]
  [extra_headers]
  Distribution = Ubuntu
  [federation]
  [fernet_receipts]
  [fernet_tokens]
  [healthcheck]
  [identity]
  [identity_mapping]
  [jwt_tokens]
  [ldap]
  [memcache]
  [oauth1]
  [oslo_messaging_amqp]
  [oslo_messaging_kafka]
  [oslo_messaging_notifications]
  [oslo_messaging_rabbit]
  [oslo_middleware]
  [oslo_policy]
  [policy]
  [profiler]
  [receipt]
  [resource]
  [revoke]
  [role]
  [saml]
  [security_compliance]
  [shadow_users]
  [token]
  provider = fernet
  caching = true
  driver = keystone.token.persistence.backends.kvs.Token
  [tokenless_auth]
  [totp]
  [trust]
  [unified_limit]
  [wsgi]

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1926836/+subscriptions