yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1834294] [NEW] python can not handle non-latin characters

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Eivinas <1834294@xxxxxxxxxxxxxxxxxx>
Date: Wed, 26 Jun 2019 07:55:29 -0000
Reply-to: Bug 1834294 <1834294@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

As there was a problem in previous openstack releases that ldap can not
handle non-latin characters. Now it is possible to describe user_tree_dn
in keystone.domain.conf with non-latin characters, but as trying to set
"user_filter" with non-latin characters getting an error. When i comment
out it works just fine.

Openstack Queens Ubuntu 18.04

/etc/keystone/domains/keystone.domain.conf
[ldap]
user_tree_dn= OU=ITSC,OU=one,DC=example,DC=com
user_filter = (memberOf=CN=Infrastruktūros skyrius,OU=Infrastruktūros skyrius,OU=ITSC,OU=one,DC=example,DC=com)


ERROR:
RESP BODY: {"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

GET call to identity for http://10.10.10.11:35357/v3/users?domain_id=fe6d7703cf524cc793f5f9d34e63ef54 used request id req-33b9cf55-6435-4a16-aa06-56e00bf16186
Request returned failure status: 500
An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-33b9cf55-6435-4a16-aa06-56e00bf16186)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/identity/v3/user.py", line 266, in take_action
    group=group,
  File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/users.py", line 136, in list
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
    return f(*args, **new_kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
    self.collection_key)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
    resp, body = self.client.get(url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/osc_lib/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request
    raise exceptions.from_response(resp, method, url)
InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-33b9cf55-6435-4a16-aa06-56e00bf16186)
clean_up ListUser: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-33b9cf55-6435-4a16-aa06-56e00bf16186)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 134, in run
    ret_val = super(OpenStackShell, self).run(argv)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 279, in run
    result = self.run_subcommand(remainder)
  File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 169, in run_subcommand
    ret_value = super(OpenStackShell, self).run_subcommand(argv)
  File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
    result = cmd.run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
    return super(Command, self).run(parsed_args)
  File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run
    column_names, data = self.take_action(parsed_args)
  File "/usr/lib/python2.7/dist-packages/openstackclient/identity/v3/user.py", line 266, in take_action
    group=group,
  File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator
    return wrapped(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/users.py", line 136, in list
    **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
    return f(*args, **new_kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
    self.collection_key)
  File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
    resp, body = self.client.get(url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/osc_lib/session.py", line 40, in request
    resp = super(TimingSession, self).request(url, method, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request
    raise exceptions.from_response(resp, method, url)
InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-33b9cf55-6435-4a16-aa06-56e00bf16186)

/var/log/keystone/keystone.log

(keystone.common.wsgi): 2019-06-26 07:47:51,312 ERROR 'ascii' codec can't decode byte 0xc5 in position 24: ordinal not in range(128)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
    result = method(req, **params)
  File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 126, in wrapper
    return f(self, request, filters, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/controllers.py", line 71, in list_users
    domain_scope=domain, hints=hints
  File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 116, in wrapped
    __ret_val = __f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 416, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 426, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 1074, in list_users
    ref_list = self._handle_shadow_and_local_users(driver, hints)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 1047, in _handle_shadow_and_local_users
    res = driver.list_users(hints)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/core.py", line 87, in list_users
    return self.user.get_all_filtered(hints)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/core.py", line 327, in get_all_filtered
    for user in self.get_all(query, hints)]
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/core.py", line 319, in get_all
    hints=hints)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/common.py", line 1850, in get_all
    return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/common.py", line 1552, in get_all
    for x in self._ldap_get_all(hints, ldap_filter)]
  File "/usr/lib/python2.7/dist-packages/keystone/common/driver_hints.py", line 42, in wrapper
    return f(self, hints, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/common.py", line 1487, in _ldap_get_all
    self.id_attr)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 24: ordinal not in range(128)

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1834294

Title:
  python can not handle non-latin characters

Status in OpenStack Identity (keystone):
  New

Bug description:
  As there was a problem in previous openstack releases that ldap can
  not handle non-latin characters. Now it is possible to describe
  user_tree_dn in keystone.domain.conf with non-latin characters, but as
  trying to set "user_filter" with non-latin characters getting an
  error. When i comment out it works just fine.

  Openstack Queens Ubuntu 18.04

  /etc/keystone/domains/keystone.domain.conf
  [ldap]
  user_tree_dn= OU=ITSC,OU=one,DC=example,DC=com
  user_filter = (memberOf=CN=Infrastruktūros skyrius,OU=Infrastruktūros skyrius,OU=ITSC,OU=one,DC=example,DC=com)

  
  ERROR:
  RESP BODY: {"error": {"message": "An unexpected error prevented the server from fulfilling your request.", "code": 500, "title": "Internal Server Error"}}

  GET call to identity for http://10.10.10.11:35357/v3/users?domain_id=fe6d7703cf524cc793f5f9d34e63ef54 used request id req-33b9cf55-6435-4a16-aa06-56e00bf16186
  Request returned failure status: 500
  An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-33b9cf55-6435-4a16-aa06-56e00bf16186)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
      result = cmd.run(parsed_args)
    File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
      return super(Command, self).run(parsed_args)
    File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run
      column_names, data = self.take_action(parsed_args)
    File "/usr/lib/python2.7/dist-packages/openstackclient/identity/v3/user.py", line 266, in take_action
      group=group,
    File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator
      return wrapped(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/users.py", line 136, in list
      **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
      return f(*args, **new_kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
      self.collection_key)
    File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
      resp, body = self.client.get(url, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
      return self.request(url, 'GET', **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
      resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
      return self.session.request(url, method, **kwargs)
    File "/usr/lib/python2.7/dist-packages/osc_lib/session.py", line 40, in request
      resp = super(TimingSession, self).request(url, method, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request
      raise exceptions.from_response(resp, method, url)
  InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-33b9cf55-6435-4a16-aa06-56e00bf16186)
  clean_up ListUser: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-33b9cf55-6435-4a16-aa06-56e00bf16186)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 134, in run
      ret_val = super(OpenStackShell, self).run(argv)
    File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 279, in run
      result = self.run_subcommand(remainder)
    File "/usr/lib/python2.7/dist-packages/osc_lib/shell.py", line 169, in run_subcommand
      ret_value = super(OpenStackShell, self).run_subcommand(argv)
    File "/usr/lib/python2.7/dist-packages/cliff/app.py", line 400, in run_subcommand
      result = cmd.run(parsed_args)
    File "/usr/lib/python2.7/dist-packages/osc_lib/command/command.py", line 41, in run
      return super(Command, self).run(parsed_args)
    File "/usr/lib/python2.7/dist-packages/cliff/display.py", line 116, in run
      column_names, data = self.take_action(parsed_args)
    File "/usr/lib/python2.7/dist-packages/openstackclient/identity/v3/user.py", line 266, in take_action
      group=group,
    File "/usr/lib/python2.7/dist-packages/debtcollector/renames.py", line 43, in decorator
      return wrapped(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/users.py", line 136, in list
      **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
      return f(*args, **new_kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 397, in list
      self.collection_key)
    File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 125, in _list
      resp, body = self.client.get(url, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 304, in get
      return self.request(url, 'GET', **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 463, in request
      resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 189, in request
      return self.session.request(url, method, **kwargs)
    File "/usr/lib/python2.7/dist-packages/osc_lib/session.py", line 40, in request
      resp = super(TimingSession, self).request(url, method, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystoneauth1/session.py", line 737, in request
      raise exceptions.from_response(resp, method, url)
  InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-33b9cf55-6435-4a16-aa06-56e00bf16186)

  /var/log/keystone/keystone.log

  (keystone.common.wsgi): 2019-06-26 07:47:51,312 ERROR 'ascii' codec can't decode byte 0xc5 in position 24: ordinal not in range(128)
  Traceback (most recent call last):
    File "/usr/lib/python2.7/dist-packages/keystone/common/wsgi.py", line 226, in __call__
      result = method(req, **params)
    File "/usr/lib/python2.7/dist-packages/keystone/common/controller.py", line 126, in wrapper
      return f(self, request, filters, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/controllers.py", line 71, in list_users
      domain_scope=domain, hints=hints
    File "/usr/lib/python2.7/dist-packages/keystone/common/manager.py", line 116, in wrapped
      __ret_val = __f(*args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 416, in wrapper
      return f(self, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 426, in wrapper
      return f(self, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 1074, in list_users
      ref_list = self._handle_shadow_and_local_users(driver, hints)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/core.py", line 1047, in _handle_shadow_and_local_users
      res = driver.list_users(hints)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/core.py", line 87, in list_users
      return self.user.get_all_filtered(hints)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/core.py", line 327, in get_all_filtered
      for user in self.get_all(query, hints)]
    File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/core.py", line 319, in get_all
      hints=hints)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/common.py", line 1850, in get_all
      return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/common.py", line 1552, in get_all
      for x in self._ldap_get_all(hints, ldap_filter)]
    File "/usr/lib/python2.7/dist-packages/keystone/common/driver_hints.py", line 42, in wrapper
      return f(self, hints, *args, **kwargs)
    File "/usr/lib/python2.7/dist-packages/keystone/identity/backends/ldap/common.py", line 1487, in _ldap_get_all
      self.id_attr)
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 24: ordinal not in range(128)

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1834294/+subscriptions