yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1953627] [NEW] [ldappool] Downing one of the configured LDAP servers causes a persistent failure

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Grzegorz Grasza <1953627@xxxxxxxxxxxxxxxxxx>
Date: Wed, 08 Dec 2021 14:33:17 -0000
Reply-to: Bug 1953627 <1953627@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

If a server disconnects after a pooled connection is created, it fails
withe an ldap.TIMEOUT which is Unhandled.

The ReconnectLDAPObject used by ldappool only catches the
ldap.SERVER_DOWN exception in _apply_method_s which is applied to
synchronous methods, whereas ldap.TIMEOUT is properly caught only during
the initial connection.


To test this I did the following:

1. Create a new interface with a new IP address for the ldap server

2. set this as the first server in the url list and set
pool_connection_timeout in the domain [ldap] configuration

3. Do a couple of:

  time openstack user list --domain=Users

to fill the connections in the pools in all WSGI processes

4. remove the created IP address from the interface

5. try an openstack user list again

Result:

the pooled connection consistently fails with:

gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: CRITICAL keystone [None req-7ad1997a-e23e-436b-a9ad-72f68a52abf8 demo admin] Unhandled error: ldap.TIMEOUT
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone Traceback (most recent call last):         
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2091, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.wsgi_app(environ, start_response)                
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__    
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.app(environ, start_response)                                                               
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__         
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)                                         
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func               
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)                                                      
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/base.py", line 124, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = req.get_response(self.application)                                                      
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send         
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(                              
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)                                            
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 143, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return resp(environ, start_response)                                                            
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__                
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)                                                         
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func         
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)                  
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/base.py", line 124, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = req.get_response(self.application)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send             
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(    
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application    
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)                                            
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__                 
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)                                  
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func                
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/osprofiler/web.py", line 112, in __call__     
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return request.get_response(self.application)                                                
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send   
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(                                      
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)                                       
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__     
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)                                             
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func    
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)                                                                
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = req.get_response(self.application)                   
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(                      
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application           gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)                     
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py",[50/1978]
 in __call__
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __c
all__
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = req.get_response(self._app)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/werkzeug/middleware/dispatcher.py", line 78, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2076, in wsgi_app
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = self.handle_exception(e)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   [Previous line repeated 27 more times]
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2073, in wsgi_app
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = self.full_dispatch_request()
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1518, in full_dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     rv = self.handle_user_exception(e)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   [Previous line repeated 27 more times]
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1516, in full_dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     rv = self.dispatch_request()
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1502, in dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 467, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = resource(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/views.py", line 84, in view
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = meth(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/api/users.py", line 183, in get
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self._list_users()
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/api/users.py", line 214, in _list_users
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     refs = PROVIDERS.identity_api.list_users(
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     __ret_val = __f(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return f(self, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return f(self, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 1108, in list_users
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     ref_list = self._handle_shadow_and_local_users(driver, hints)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 1091, in _handle_shadow_and_local_users
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return driver.list_users(hints) + fed_res
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 85, in list_users
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.user.get_all_filtered(hints)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 328, in get_all_filtered
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     for user in self.get_all(query, hints)]
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 319, in get_all
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     objs = super(UserApi, self).get_all(ldap_filter=ldap_filter,
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1934, in get_all
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1622, in get_all
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     for x in self._ldap_get_all(hints, ldap_filter)]
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/common/driver_hints.py", line 42, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return f(self, hints, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1572, in _ldap_get_all
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     res = conn.search_s(self.tree_dn,
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 985, in search_s
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     ldap_result = self.conn.search_s(base, scope, filterstr,
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 679, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return func(self, conn, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 813, in search_s
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return conn.search_s(base, scope, filterstr, attrlist,
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 631, in search_s
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 1030, in search_ext_s
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 968, in _apply_method_s
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return func(self,*args,**kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 625, in search_ext_s
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.result(msgid,all=1,timeout=timeout)[1]
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 535, in result
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp_type, resp_data, resp_msgid = self.result2(msgid,all,timeout)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 539, in result2
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all,timeout)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 543, in result3
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp_type, resp_data, resp_msgid, decoded_resp_ctrls, retoid, retval = self.result4(
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 553, in result4
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 128, in _ldap_call
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     result = func(*args,**kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone ldap.TIMEOUT 
gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone

** Affects: keystone
     Importance: Undecided
         Status: New


** Tags: ldap

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1953627

Title:
  [ldappool] Downing one of the configured LDAP servers causes a
  persistent failure

Status in OpenStack Identity (keystone):
  New

Bug description:
  If a server disconnects after a pooled connection is created, it fails
  withe an ldap.TIMEOUT which is Unhandled.

  The ReconnectLDAPObject used by ldappool only catches the
  ldap.SERVER_DOWN exception in _apply_method_s which is applied to
  synchronous methods, whereas ldap.TIMEOUT is properly caught only
  during the initial connection.

  
  To test this I did the following:

  1. Create a new interface with a new IP address for the ldap server

  2. set this as the first server in the url list and set
  pool_connection_timeout in the domain [ldap] configuration

  3. Do a couple of:

    time openstack user list --domain=Users

  to fill the connections in the pools in all WSGI processes

  4. remove the created IP address from the interface

  5. try an openstack user list again

  Result:

  the pooled connection consistently fails with:

  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: CRITICAL keystone [None req-7ad1997a-e23e-436b-a9ad-72f68a52abf8 demo admin] Unhandled error: ldap.TIMEOUT
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone Traceback (most recent call last):         
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2091, in __call__
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.wsgi_app(environ, start_response)                
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__    
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.app(environ, start_response)                                                               
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__         
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)                                         
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func               
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)                                                      
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/base.py", line 124, in __call__
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = req.get_response(self.application)                                                      
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send         
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(                              
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)                                            
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 143, in __call__
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return resp(environ, start_response)                                                            
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__                
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)                                                         
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func         
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)                  
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/base.py", line 124, in __call__
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = req.get_response(self.application)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send             
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(    
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application    
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)                                            
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__                 
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)                                  
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func                
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/osprofiler/web.py", line 112, in __call__     
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return request.get_response(self.application)                                                
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send   
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(                                      
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)                                       
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__     
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)                                             
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func    
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)                                                                
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = req.get_response(self.application)                   
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(                      
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application           gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)                     
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py",[50/1978]
   in __call__
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.app(environ, start_response)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.func(req, *args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __c
  all__
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = req.get_response(self._app)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     status, headers, app_iter = self.call_application(
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     app_iter = application(self.environ, start_response)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/werkzeug/middleware/dispatcher.py", line 78, in __call__
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return app(environ, start_response)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2076, in wsgi_app
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = self.handle_exception(e)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   [Previous line repeated 27 more times]
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2073, in wsgi_app
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     response = self.full_dispatch_request()
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1518, in full_dispatch_request
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     rv = self.handle_user_exception(e)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return original_handler(e)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   [Previous line repeated 27 more times]
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1516, in full_dispatch_request
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     rv = self.dispatch_request()
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1502, in dispatch_request
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 467, in wrapper
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = resource(*args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask/views.py", line 84, in view
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp = meth(*args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/api/users.py", line 183, in get
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self._list_users()
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/api/users.py", line 214, in _list_users
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     refs = PROVIDERS.identity_api.list_users(
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     __ret_val = __f(*args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return f(self, *args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return f(self, *args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 1108, in list_users
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     ref_list = self._handle_shadow_and_local_users(driver, hints)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/core.py", line 1091, in _handle_shadow_and_local_users
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return driver.list_users(hints) + fed_res
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 85, in list_users
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.user.get_all_filtered(hints)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 328, in get_all_filtered
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     for user in self.get_all(query, hints)]
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 319, in get_all
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     objs = super(UserApi, self).get_all(ldap_filter=ldap_filter,
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1934, in get_all
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1622, in get_all
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     for x in self._ldap_get_all(hints, ldap_filter)]
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/common/driver_hints.py", line 42, in wrapper
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return f(self, hints, *args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1572, in _ldap_get_all
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     res = conn.search_s(self.tree_dn,
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 985, in search_s
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     ldap_result = self.conn.search_s(base, scope, filterstr,
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 679, in wrapper
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return func(self, conn, *args, **kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 813, in search_s
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return conn.search_s(base, scope, filterstr, attrlist,
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 631, in search_s
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 1030, in search_ext_s
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 968, in _apply_method_s
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return func(self,*args,**kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 625, in search_ext_s
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     return self.result(msgid,all=1,timeout=timeout)[1]
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 535, in result
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp_type, resp_data, resp_msgid = self.result2(msgid,all,timeout)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 539, in result2
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all,timeout)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 543, in result3
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     resp_type, resp_data, resp_msgid, decoded_resp_ctrls, retoid, retval = self.result4(
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 553, in result4
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone   File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 128, in _ldap_call
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone     result = func(*args,**kwargs)
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone ldap.TIMEOUT 
  gru 08 14:21:01 ggrasza-ubuntu2 devstack@keystone.service[1270797]: ERROR keystone

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1953627/+subscriptions