yahoo-eng-team team mailing list archive
-
yahoo-eng-team team
-
Mailing list archive
-
Message #77541
[Bug 1820333] Re: ldap search should not encode attributes
** Description changed:
+ Listing user fails with LDAP backend
+ ------------------------------------
+
$ openstack user list --debug --domain userdomain
Request returned failure status: 400
('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
Traceback (most recent call last):
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
- result = cmd.run(parsed_args)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
- return super(Command, self).run(parsed_args)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
- column_names, data = self.take_action(parsed_args)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
- group=group,
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
- return wrapped(*args, **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
- **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
- return f(*args, **new_kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
- list_resp = self._list(url_query, self.collection_key)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
- resp, body = self.client.get(url, **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
- return self.request(url, 'GET', **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
- resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
- return self.session.request(url, method, **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
- raise exceptions.from_response(resp, method, url)
- keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
+ result = cmd.run(parsed_args)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
+ return super(Command, self).run(parsed_args)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
+ column_names, data = self.take_action(parsed_args)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
+ group=group,
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
+ return wrapped(*args, **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
+ **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
+ return f(*args, **new_kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
+ list_resp = self._list(url_query, self.collection_key)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
+ resp, body = self.client.get(url, **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
+ return self.request(url, 'GET', **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
+ resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
+ return self.session.request(url, method, **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
+ raise exceptions.from_response(resp, method, url)
+ keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
clean_up ListUser: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
Traceback (most recent call last):
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
- ret_val = super(OpenStackShell, self).run(argv)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 281, in run
- result = self.run_subcommand(remainder)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
- ret_value = super(OpenStackShell, self).run_subcommand(argv)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
- result = cmd.run(parsed_args)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
- return super(Command, self).run(parsed_args)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
- column_names, data = self.take_action(parsed_args)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
- group=group,
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
- return wrapped(*args, **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
- **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
- return f(*args, **new_kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
- list_resp = self._list(url_query, self.collection_key)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
- resp, body = self.client.get(url, **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
- return self.request(url, 'GET', **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
- resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
- return self.session.request(url, method, **kwargs)
- File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
- raise exceptions.from_response(resp, method, url)
- keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
+ ret_val = super(OpenStackShell, self).run(argv)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 281, in run
+ result = self.run_subcommand(remainder)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
+ ret_value = super(OpenStackShell, self).run_subcommand(argv)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
+ result = cmd.run(parsed_args)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
+ return super(Command, self).run(parsed_args)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
+ column_names, data = self.take_action(parsed_args)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
+ group=group,
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
+ return wrapped(*args, **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
+ **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
+ return f(*args, **new_kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
+ list_resp = self._list(url_query, self.collection_key)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
+ resp, body = self.client.get(url, **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
+ return self.request(url, 'GET', **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
+ resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
+ return self.session.request(url, method, **kwargs)
+ File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
+ raise exceptions.from_response(resp, method, url)
+ keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
END return value: 1
/var/log/keystone/keystone.log
-
+ ------------------------------
(keystone.common.wsgi): 2019-03-15 15:26:15,385 ERROR ('attrs_from_List(): expected string in list', b'mail')
Traceback (most recent call last):
- File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 148, in __call__
- result = method(req, **params)
- File "/usr/lib/python3/dist-packages/keystone/common/controller.py", line 103, in wrapper
- return f(self, request, filters, **kwargs)
- File "/usr/lib/python3/dist-packages/keystone/identity/controllers.py", line 71, in list_users
- domain_scope=domain, hints=hints
- File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped
- __ret_val = __f(*args, **kwargs)
- File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 416, in wrapper
- return f(self, *args, **kwargs)
- File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 426, in wrapper
- return f(self, *args, **kwargs)
- File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1061, in list_users
- ref_list = self._handle_shadow_and_local_users(driver, hints)
- File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1044, in _handle_shadow_and_local_users
- return driver.list_users(hints) + fed_res
- File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 87, in list_users
- return self.user.get_all_filtered(hints)
- File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 327, in get_all_filtered
- for user in self.get_all(query, hints)]
- File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 319, in get_all
- hints=hints)
- File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1888, in get_all
- return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
- File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1590, in get_all
- for x in self._ldap_get_all(hints, ldap_filter)]
- File "/usr/lib/python3/dist-packages/keystone/common/driver_hints.py", line 42, in wrapper
- return f(self, hints, *args, **kwargs)
- File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1543, in _ldap_get_all
- attrs)
- File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 976, in search_s
- attrlist_utf8, attrsonly)
- File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 654, in wrapper
- return func(self, conn, *args, **kwargs)
- File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 803, in search_s
- attrsonly)
- File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 858, in search_s
- return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
- File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1264, in search_ext_s
- return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
- File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1202, in _apply_method_s
- return func(self,*args,**kwargs)
- File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 851, in search_ext_s
- msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
- File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 847, in search_ext
- timeout,sizelimit,
- File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 319, in _ldap_call
- result = func(*args,**kwargs)
+ File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 148, in __call__
+ result = method(req, **params)
+ File "/usr/lib/python3/dist-packages/keystone/common/controller.py", line 103, in wrapper
+ return f(self, request, filters, **kwargs)
+ File "/usr/lib/python3/dist-packages/keystone/identity/controllers.py", line 71, in list_users
+ domain_scope=domain, hints=hints
+ File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped
+ __ret_val = __f(*args, **kwargs)
+ File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 416, in wrapper
+ return f(self, *args, **kwargs)
+ File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 426, in wrapper
+ return f(self, *args, **kwargs)
+ File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1061, in list_users
+ ref_list = self._handle_shadow_and_local_users(driver, hints)
+ File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1044, in _handle_shadow_and_local_users
+ return driver.list_users(hints) + fed_res
+ File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 87, in list_users
+ return self.user.get_all_filtered(hints)
+ File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 327, in get_all_filtered
+ for user in self.get_all(query, hints)]
+ File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 319, in get_all
+ hints=hints)
+ File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1888, in get_all
+ return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
+ File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1590, in get_all
+ for x in self._ldap_get_all(hints, ldap_filter)]
+ File "/usr/lib/python3/dist-packages/keystone/common/driver_hints.py", line 42, in wrapper
+ return f(self, hints, *args, **kwargs)
+ File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1543, in _ldap_get_all
+ attrs)
+ File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 976, in search_s
+ attrlist_utf8, attrsonly)
+ File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 654, in wrapper
+ return func(self, conn, *args, **kwargs)
+ File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 803, in search_s
+ attrsonly)
+ File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 858, in search_s
+ return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
+ File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1264, in search_ext_s
+ return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
+ File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1202, in _apply_method_s
+ return func(self,*args,**kwargs)
+ File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 851, in search_ext_s
+ msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
+ File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 847, in search_ext
+ timeout,sizelimit,
+ File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 319, in _ldap_call
+ result = func(*args,**kwargs)
TypeError: ('attrs_from_List(): expected string in list', b'mail')
+ ---------------------------------------------
- In search_s() we're still encoding attrlist:
+ In search_s() we're still encoding attrlist (note similar behavior in
+ paged_search_s):
attrlist_utf8 = list(map(utf8_encode, attrlist))
Looking closer at the attribute list these all appear to be attribute
- names:
+ names and that also appears to be how LDAP searches generally work; they
+ specify attribute names they want to return, not values:
[b'enabled', b'sn', b'userPassword', b'cn', b'description', b'mail']
In Python 3 (and Python2 with bytes_mode=False) python-ldap no longer
allows bytes for some fields (DNs, RDNs, attribute names, queries).
- Instead, text values are represented as str, the Unicode text type. A
- prior patch to Keystone's LDAP backend (see commit
+ Instead, text values are represented as str, the Unicode text type.
+
+ A prior patch to Keystone's LDAP backend (see commit
eca0829c4c65e6b64f08023ce2d5a55dc329248f) enabled this support but
- missed the above line of code.
-
+ missed the above lines of code.
Changing the above line of code to not utf8 encode the attrlist fixes
the problem for me.
** Also affects: keystone (Ubuntu)
Importance: Undecided
Status: New
** Also affects: cloud-archive
Importance: Undecided
Status: New
** Changed in: cloud-archive
Status: New => Triaged
** Changed in: cloud-archive
Importance: Undecided => Critical
** Changed in: keystone (Ubuntu)
Status: New => Triaged
** Changed in: keystone (Ubuntu)
Importance: Undecided => Critical
** Summary changed:
- ldap search should not encode attributes
+ [SRU] ldap search should not encode attributes
** Description changed:
- Listing user fails with LDAP backend
- ------------------------------------
+ [Impact]
+
+ Listing user fails with LDAP backend fails
+ ------------------------------------------
$ openstack user list --debug --domain userdomain
Request returned failure status: 400
('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
Traceback (most recent call last):
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
result = cmd.run(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
column_names, data = self.take_action(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
group=group,
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
return wrapped(*args, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
**kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
return f(*args, **new_kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
list_resp = self._list(url_query, self.collection_key)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
resp, body = self.client.get(url, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
return self.request(url, 'GET', **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
return self.session.request(url, method, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
clean_up ListUser: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
Traceback (most recent call last):
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
ret_val = super(OpenStackShell, self).run(argv)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 281, in run
result = self.run_subcommand(remainder)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
ret_value = super(OpenStackShell, self).run_subcommand(argv)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
result = cmd.run(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
column_names, data = self.take_action(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
group=group,
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
return wrapped(*args, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
**kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
return f(*args, **new_kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
list_resp = self._list(url_query, self.collection_key)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
resp, body = self.client.get(url, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
return self.request(url, 'GET', **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
return self.session.request(url, method, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
END return value: 1
-
/var/log/keystone/keystone.log
------------------------------
(keystone.common.wsgi): 2019-03-15 15:26:15,385 ERROR ('attrs_from_List(): expected string in list', b'mail')
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 148, in __call__
result = method(req, **params)
File "/usr/lib/python3/dist-packages/keystone/common/controller.py", line 103, in wrapper
return f(self, request, filters, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/controllers.py", line 71, in list_users
domain_scope=domain, hints=hints
File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped
__ret_val = __f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 416, in wrapper
return f(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 426, in wrapper
return f(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1061, in list_users
ref_list = self._handle_shadow_and_local_users(driver, hints)
File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1044, in _handle_shadow_and_local_users
return driver.list_users(hints) + fed_res
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 87, in list_users
return self.user.get_all_filtered(hints)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 327, in get_all_filtered
for user in self.get_all(query, hints)]
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 319, in get_all
hints=hints)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1888, in get_all
return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1590, in get_all
for x in self._ldap_get_all(hints, ldap_filter)]
File "/usr/lib/python3/dist-packages/keystone/common/driver_hints.py", line 42, in wrapper
return f(self, hints, *args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1543, in _ldap_get_all
attrs)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 976, in search_s
attrlist_utf8, attrsonly)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 654, in wrapper
return func(self, conn, *args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 803, in search_s
attrsonly)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 858, in search_s
return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1264, in search_ext_s
return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1202, in _apply_method_s
return func(self,*args,**kwargs)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 851, in search_ext_s
msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 847, in search_ext
timeout,sizelimit,
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 319, in _ldap_call
result = func(*args,**kwargs)
TypeError: ('attrs_from_List(): expected string in list', b'mail')
---------------------------------------------
In search_s() we're still encoding attrlist (note similar behavior in
paged_search_s):
attrlist_utf8 = list(map(utf8_encode, attrlist))
Looking closer at the attribute list these all appear to be attribute
names and that also appears to be how LDAP searches generally work; they
specify attribute names they want to return, not values:
[b'enabled', b'sn', b'userPassword', b'cn', b'description', b'mail']
In Python 3 (and Python2 with bytes_mode=False) python-ldap no longer
allows bytes for some fields (DNs, RDNs, attribute names, queries).
Instead, text values are represented as str, the Unicode text type.
A prior patch to Keystone's LDAP backend (see commit
eca0829c4c65e6b64f08023ce2d5a55dc329248f) enabled this support but
missed the above lines of code.
Changing the above line of code to not utf8 encode the attrlist fixes
the problem for me.
+
+ [Test Case]
+
+ Run charm-keystone-ldap functional tests for OpenStack Rocky or above.
+ Upstream unit tests are also run.
+
+ [Regression Potential]
+ The only regression potential would be for PY2 code paths. PY3 code paths never worked for keystone's LDAP backend. The approach to the patch have purposefully minimized amount of code required and therefore regression potential for PY2. Note that Rocky for Ubuntu supports PY2 but as of Stein Ubuntu has dropped PY2 support.
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1820333
Title:
[SRU] ldap search should not encode attributes
Status in Ubuntu Cloud Archive:
Triaged
Status in OpenStack Identity (keystone):
New
Status in keystone package in Ubuntu:
Triaged
Bug description:
[Impact]
Listing user fails with LDAP backend fails
------------------------------------------
$ openstack user list --debug --domain userdomain
Request returned failure status: 400
('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
Traceback (most recent call last):
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
result = cmd.run(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
column_names, data = self.take_action(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
group=group,
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
return wrapped(*args, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
**kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
return f(*args, **new_kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
list_resp = self._list(url_query, self.collection_key)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
resp, body = self.client.get(url, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
return self.request(url, 'GET', **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
return self.session.request(url, method, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
clean_up ListUser: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
Traceback (most recent call last):
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 136, in run
ret_val = super(OpenStackShell, self).run(argv)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 281, in run
result = self.run_subcommand(remainder)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/shell.py", line 176, in run_subcommand
ret_value = super(OpenStackShell, self).run_subcommand(argv)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/app.py", line 401, in run_subcommand
result = cmd.run(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
return super(Command, self).run(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/cliff/display.py", line 116, in run
column_names, data = self.take_action(parsed_args)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/openstackclient/identity/v3/user.py", line 266, in take_action
group=group,
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/debtcollector/renames.py", line 43, in decorator
return wrapped(*args, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/v3/users.py", line 136, in list
**kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func
return f(*args, **new_kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 444, in list
list_resp = self._list(url_query, self.collection_key)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list
resp, body = self.client.get(url, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 351, in get
return self.request(url, 'GET', **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 510, in request
resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 213, in request
return self.session.request(url, method, **kwargs)
File "/home/ubuntu/charm-test-infra/.tox/clients/lib/python3.6/site-packages/keystoneauth1/session.py", line 869, in request
raise exceptions.from_response(resp, method, url)
keystoneauth1.exceptions.http.BadRequest: ('attrs_from_List(): expected string in list', b'mail') (HTTP 400) (Request-ID: req-914f8010-3ed2-4200-a394-5b1bc5158b98)
END return value: 1
/var/log/keystone/keystone.log
------------------------------
(keystone.common.wsgi): 2019-03-15 15:26:15,385 ERROR ('attrs_from_List(): expected string in list', b'mail')
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/keystone/common/wsgi.py", line 148, in __call__
result = method(req, **params)
File "/usr/lib/python3/dist-packages/keystone/common/controller.py", line 103, in wrapper
return f(self, request, filters, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/controllers.py", line 71, in list_users
domain_scope=domain, hints=hints
File "/usr/lib/python3/dist-packages/keystone/common/manager.py", line 116, in wrapped
__ret_val = __f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 416, in wrapper
return f(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 426, in wrapper
return f(self, *args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1061, in list_users
ref_list = self._handle_shadow_and_local_users(driver, hints)
File "/usr/lib/python3/dist-packages/keystone/identity/core.py", line 1044, in _handle_shadow_and_local_users
return driver.list_users(hints) + fed_res
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 87, in list_users
return self.user.get_all_filtered(hints)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 327, in get_all_filtered
for user in self.get_all(query, hints)]
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/core.py", line 319, in get_all
hints=hints)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1888, in get_all
return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1590, in get_all
for x in self._ldap_get_all(hints, ldap_filter)]
File "/usr/lib/python3/dist-packages/keystone/common/driver_hints.py", line 42, in wrapper
return f(self, hints, *args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 1543, in _ldap_get_all
attrs)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 976, in search_s
attrlist_utf8, attrsonly)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 654, in wrapper
return func(self, conn, *args, **kwargs)
File "/usr/lib/python3/dist-packages/keystone/identity/backends/ldap/common.py", line 803, in search_s
attrsonly)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 858, in search_s
return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1264, in search_ext_s
return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1202, in _apply_method_s
return func(self,*args,**kwargs)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 851, in search_ext_s
msgid = self.search_ext(base,scope,filterstr,attrlist,attrsonly,serverctrls,clientctrls,timeout,sizelimit)
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 847, in search_ext
timeout,sizelimit,
File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 319, in _ldap_call
result = func(*args,**kwargs)
TypeError: ('attrs_from_List(): expected string in list', b'mail')
---------------------------------------------
In search_s() we're still encoding attrlist (note similar behavior in
paged_search_s):
attrlist_utf8 = list(map(utf8_encode, attrlist))
Looking closer at the attribute list these all appear to be attribute
names and that also appears to be how LDAP searches generally work;
they specify attribute names they want to return, not values:
[b'enabled', b'sn', b'userPassword', b'cn', b'description', b'mail']
In Python 3 (and Python2 with bytes_mode=False) python-ldap no longer
allows bytes for some fields (DNs, RDNs, attribute names, queries).
Instead, text values are represented as str, the Unicode text type.
A prior patch to Keystone's LDAP backend (see commit
eca0829c4c65e6b64f08023ce2d5a55dc329248f) enabled this support but
missed the above lines of code.
Changing the above line of code to not utf8 encode the attrlist fixes
the problem for me.
[Test Case]
Run charm-keystone-ldap functional tests for OpenStack Rocky or above.
Upstream unit tests are also run.
[Regression Potential]
The only regression potential would be for PY2 code paths. PY3 code paths never worked for keystone's LDAP backend. The approach to the patch have purposefully minimized amount of code required and therefore regression potential for PY2. Note that Rocky for Ubuntu supports PY2 but as of Stein Ubuntu has dropped PY2 support.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-archive/+bug/1820333/+subscriptions
References
