yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1801873] [NEW] Unable to delete domains when users was managed by LDAP back-end

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Izak A Eygelaar <1801873@xxxxxxxxxxxxxxxxxx>
Date: Tue, 06 Nov 2018 07:41:23 -0000
Reply-to: Bug 1801873 <1801873@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

Problem description:
We can create domains and assign users from LDAP back-end to the domains and create projects inside of the domains and also assign users out of LDAP access to the project. When we try to delete the domains that made use of the LDAP domain configuration to manage users, the domain deletion fails with a 500 server error.

Reproducing error:
Flow for creating domains:
1.) Create new domain
2.) Add domain configuration for LDAP config to domain
3.) Assign user out of LDAP access to the domain
4.) Create project inside of Domain and assign user access to the relevant project(s)

Flow for deleting domains:
1.) Revoke user access to the projects
2.) Remove all projects assigned in the Domain
3.) Revoke user access on domain level
4.) Remove the LDAP domain configuration for the domain
5.) Set domain status to disabled
6.) Remove domain (fails with server 500 error)

LDAP domain configuration added making use of the API call:
{
    "config": {
        "identity": {
            "driver": "ldap"
        },
        "ldap": {
            "user_mail_attribute": "mail",
            "url": "ldaps://ldap.x.x.x",
            "user_id_attribute": "mail",
            "query_scope": "sub",
            "user_description_attribute": "description",
            "user_enabled_mask": "2",
            "user_enabled_default": "512",
            "user_enabled_invert": "true",
            "user_enabled_attribute": "enabled",
            "user_name_attribute": "mail",
            "user_pass_attribute": "userPassword",
            "user_objectclass": "inetOrgPerson",
            "user_tree_dn": "ou=Openstack,dc=x,dc=x,dc=x"
        }
    }
}

Keystone config:
[identity]
domain_configurations_from_database = True
domain_specific_drivers_enabled = True
driver = sql

[ldap]
group_allow_create = False
group_allow_delete = False
group_allow_update = False
query_scope = sub
user_allow_create = False
user_allow_delete = False
user_allow_update = False

After some further investigation, it looks like remote LDAP users are
then also created "locally" on keystone database in the user tables,
which maps the users to the domains, and as a result when we remove the
LDAP domain configuration it keeps the "locally mapped" users in the
Keystone DB, which looks like Openstack performs a sanity check and
checks if users are still assigned to domains, before deleting domains.

Keystone DB user table:
+------------------------------------------------------------------+-------------------------------------------------------+---------+----------------------------------+---------------------+----------------+----------------------------------+

| id | extra | enabled | default_project_id | created_at |
last_active_at | domain_id |

+------------------------------------------------------------------+-------------------------------------------------------+---------+----------------------------------+---------------------+----------------+----------------------------------+

| 0ed65766eef533144508267d781de2ce70c62241a85ced00b9e07d0fc1a83483 |
{"email": "example@xxxxxxxxxxx"} | 1 | NULL | 2018-10-29 16:57:06 | NULL
| 28226b751b704ed6b0040016ec14d301 |

| 10adec00e356f1cd5d0470ac8dcc9148867c3913142d6d317621998aaaefb06a |
{"email": "example2@xxxxxxxxxxx", "description": " "} | 1 | NULL |
2018-10-03 20:13:17 | NULL | 29e6ee57de1f4468821607f33aa69d9a |

| 1b92cfe2d0add50a8b1b04ba532fdb1a467adebcaa6891addcc2fa5263e565a6 |
{"email": "example3@xxxxxxxxxxx"} | 1 | NULL | 2018-10-29 16:55:10 |
NULL | 1dca35888d344a3e8c5b69604ba92806 |

| 1d12c86bafc10dd8aa6da726ec85ac7a2351957da4cda3cc7ad19863ed9bd94d |
{"email": "example4@xxxxxxxxxxx"} | 1 | NULL | 2018-10-31 06:48:34 |
NULL | 2dd53ca43c3144ca86e0b4425d61668e |


Stack trace:
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi [req-ddd4c5b0-54ef-46e1-a267-d7f0c3f75106 8ae9dd0a0bbbe702ff0e0baa0dbeeb287ac6b70af521e4d99ed8bd5ba089c48f - d143c253f5ea437881a4263d3544a046 d143c253f5ea437881a4263d3544a046 -] An unexpected error prevented the server from fulfilling your request.: UnexpectedError: An unexpected error prevented the server from fulfilling your request.
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi Traceback (most recent call last):
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/wsgi.py", line 226, in __call__
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     result = method(req, **params)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/controller.py", line 82, in inner
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     return f(self, request, *args, **kwargs)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/resource/controllers.py", line 75, in delete_domain
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     domain_id, initiator=request.audit_initiator
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/resource/core.py", line 779, in delete_domain
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     self._delete_project(domain_id, initiator)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/resource/core.py", line 511, in _delete_project
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     ret = self.driver.delete_project(project_id)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/sql/core.py", line 561, in wrapper
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     'store %s') % conflict_type)
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi UnexpectedError: An unexpected error prevented the server from fulfilling your request.
2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi

Similar bugs have been reported, but they do not make use of external
authentication for users, like LDAP. For example:
https://bugs.launchpad.net/keystone/+bug/1718747

Some more env variables:
Openstack versions affected: Pike and Queens

Keystone version on Pike:
{
    "version": {
        "status": "stable",
        "updated": "2017-02-22T00:00:00Z",
        "media-types": [
            {
                "base": "application/json",
                "type": "application/vnd.openstack.identity-v3+json"
            }
        ],
        "id": "v3.8",
        "links": [
            {
                "href": "https://192.168.101.8:5000/v3/";,
                "rel": "self"
            }
        ]
    }
}

** Affects: keystone
     Importance: Undecided
         Status: New


** Tags: domains keystone ldap

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1801873

Title:
  Unable to delete domains when users was managed by LDAP back-end

Status in OpenStack Identity (keystone):
  New

Bug description:
  Problem description:
  We can create domains and assign users from LDAP back-end to the domains and create projects inside of the domains and also assign users out of LDAP access to the project. When we try to delete the domains that made use of the LDAP domain configuration to manage users, the domain deletion fails with a 500 server error.

  Reproducing error:
  Flow for creating domains:
  1.) Create new domain
  2.) Add domain configuration for LDAP config to domain
  3.) Assign user out of LDAP access to the domain
  4.) Create project inside of Domain and assign user access to the relevant project(s)

  Flow for deleting domains:
  1.) Revoke user access to the projects
  2.) Remove all projects assigned in the Domain
  3.) Revoke user access on domain level
  4.) Remove the LDAP domain configuration for the domain
  5.) Set domain status to disabled
  6.) Remove domain (fails with server 500 error)

  LDAP domain configuration added making use of the API call:
  {
      "config": {
          "identity": {
              "driver": "ldap"
          },
          "ldap": {
              "user_mail_attribute": "mail",
              "url": "ldaps://ldap.x.x.x",
              "user_id_attribute": "mail",
              "query_scope": "sub",
              "user_description_attribute": "description",
              "user_enabled_mask": "2",
              "user_enabled_default": "512",
              "user_enabled_invert": "true",
              "user_enabled_attribute": "enabled",
              "user_name_attribute": "mail",
              "user_pass_attribute": "userPassword",
              "user_objectclass": "inetOrgPerson",
              "user_tree_dn": "ou=Openstack,dc=x,dc=x,dc=x"
          }
      }
  }

  Keystone config:
  [identity]
  domain_configurations_from_database = True
  domain_specific_drivers_enabled = True
  driver = sql

  [ldap]
  group_allow_create = False
  group_allow_delete = False
  group_allow_update = False
  query_scope = sub
  user_allow_create = False
  user_allow_delete = False
  user_allow_update = False

  After some further investigation, it looks like remote LDAP users are
  then also created "locally" on keystone database in the user tables,
  which maps the users to the domains, and as a result when we remove
  the LDAP domain configuration it keeps the "locally mapped" users in
  the Keystone DB, which looks like Openstack performs a sanity check
  and checks if users are still assigned to domains, before deleting
  domains.

  Keystone DB user table:
  +------------------------------------------------------------------+-------------------------------------------------------+---------+----------------------------------+---------------------+----------------+----------------------------------+

  | id | extra | enabled | default_project_id | created_at |
  last_active_at | domain_id |

  +------------------------------------------------------------------+-------------------------------------------------------+---------+----------------------------------+---------------------+----------------+----------------------------------+

  | 0ed65766eef533144508267d781de2ce70c62241a85ced00b9e07d0fc1a83483 |
  {"email": "example@xxxxxxxxxxx"} | 1 | NULL | 2018-10-29 16:57:06 |
  NULL | 28226b751b704ed6b0040016ec14d301 |

  | 10adec00e356f1cd5d0470ac8dcc9148867c3913142d6d317621998aaaefb06a |
  {"email": "example2@xxxxxxxxxxx", "description": " "} | 1 | NULL |
  2018-10-03 20:13:17 | NULL | 29e6ee57de1f4468821607f33aa69d9a |

  | 1b92cfe2d0add50a8b1b04ba532fdb1a467adebcaa6891addcc2fa5263e565a6 |
  {"email": "example3@xxxxxxxxxxx"} | 1 | NULL | 2018-10-29 16:55:10 |
  NULL | 1dca35888d344a3e8c5b69604ba92806 |

  | 1d12c86bafc10dd8aa6da726ec85ac7a2351957da4cda3cc7ad19863ed9bd94d |
  {"email": "example4@xxxxxxxxxxx"} | 1 | NULL | 2018-10-31 06:48:34 |
  NULL | 2dd53ca43c3144ca86e0b4425d61668e |

  
  Stack trace:
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi [req-ddd4c5b0-54ef-46e1-a267-d7f0c3f75106 8ae9dd0a0bbbe702ff0e0baa0dbeeb287ac6b70af521e4d99ed8bd5ba089c48f - d143c253f5ea437881a4263d3544a046 d143c253f5ea437881a4263d3544a046 -] An unexpected error prevented the server from fulfilling your request.: UnexpectedError: An unexpected error prevented the server from fulfilling your request.
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi Traceback (most recent call last):
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/wsgi.py", line 226, in __call__
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     result = method(req, **params)
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/controller.py", line 82, in inner
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     return f(self, request, *args, **kwargs)
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/resource/controllers.py", line 75, in delete_domain
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     domain_id, initiator=request.audit_initiator
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/resource/core.py", line 779, in delete_domain
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     self._delete_project(domain_id, initiator)
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/resource/core.py", line 511, in _delete_project
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     ret = self.driver.delete_project(project_id)
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi   File "/openstack/venvs/keystone-17.1.2/lib/python2.7/site-packages/keystone/common/sql/core.py", line 561, in wrapper
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi     'store %s') % conflict_type)
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi UnexpectedError: An unexpected error prevented the server from fulfilling your request.
  2018-11-01 20:43:10.758 253 ERROR keystone.common.wsgi

  Similar bugs have been reported, but they do not make use of external
  authentication for users, like LDAP. For example:
  https://bugs.launchpad.net/keystone/+bug/1718747

  Some more env variables:
  Openstack versions affected: Pike and Queens

  Keystone version on Pike:
  {
      "version": {
          "status": "stable",
          "updated": "2017-02-22T00:00:00Z",
          "media-types": [
              {
                  "base": "application/json",
                  "type": "application/vnd.openstack.identity-v3+json"
              }
          ],
          "id": "v3.8",
          "links": [
              {
                  "href": "https://192.168.101.8:5000/v3/";,
                  "rel": "self"
              }
          ]
      }
  }

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1801873/+subscriptions
Follow ups

[Bug 1801873] Re: Unable to delete domains when users was managed by LDAP back-end
From: OpenStack Infra, 2019-03-30
[Bug 1801873] Re: Unable to delete domains when users was managed by LDAP back-end
From: Colleen Murphy, 2019-03-21