yahoo-eng-team team mailing list archive

Thread
Date
[Bug 2062247] [NEW] keystone Could not find project error when looking up valid projects

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Lukasz Zalewski <2062247@xxxxxxxxxxxxxxxxxx>
Date: Thu, 18 Apr 2024 12:40:38 -0000
Reply-to: Bug 2062247 <2062247@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Openstack version: 2023.1
Deployment tool: kayobe
OS: Ubuntu 22.04

After recent update to Antelope (2023.1) we are seeing the following
errors in the keystone.log when looking up valid/existing projects:

2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application [None ... - - default default] Could not find project: foo.: 
keystone.exception.ProjectNotFound: Could not find project: foo.
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application Traceback (most recent call last):
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     rv = self.dispatch_request()
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 467, in wrapper
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     resp = resource(*args, **kwargs)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 107, in view
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return current_app.ensure_sync(self.dispatch_request)(**kwargs)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     resp = meth(*args, **kwargs)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/projects.py", line 151, in get
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self._get_project(project_id)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/projects.py", line 107, in _get_project
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     project = PROVIDERS.resource_api.get_project(project_id)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 115, in wrapped
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     __ret_val = __f(*args, **kwargs)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return caller(func, *(extras + args), **kw)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self.get_or_create(
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     with Lock(
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self._enter()
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     generated = self._enter_create(value, createdtime)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self.creator()
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     created_value = creator(
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/resource/core.py", line 903, in get_project
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self.driver.get_project(project_id)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/resource/backends/sql.py", line 49, in get_project
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self._get_project(session, project_id).to_dict()
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/resource/backends/sql.py", line 44, in _get_project
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     raise exception.ProjectNotFound(project_id=project_id)
2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: foo.


We have traced it to a specific openstacksdk call and the issue can be reliably replicated (replace foo by a valid/existing project) by sourcing relevant admin credentials and running:
python -c 'import openstack; conn = openstack.connect(); valid_project = conn.identity.find_project("foo"); print(f"Project id is {valid_project.id}")'

The are no errors returned by the find_project call and correct
projectID is displayed, but keystone logs the above error.

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/2062247

Title:
  keystone Could not find project error when looking up valid projects

Status in OpenStack Identity (keystone):
  New

Bug description:
  Openstack version: 2023.1
  Deployment tool: kayobe
  OS: Ubuntu 22.04

  After recent update to Antelope (2023.1) we are seeing the following
  errors in the keystone.log when looking up valid/existing projects:

  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application [None ... - - default default] Could not find project: foo.: 
  keystone.exception.ProjectNotFound: Could not find project: foo.
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application Traceback (most recent call last):
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1820, in full_dispatch_request
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     rv = self.dispatch_request()
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/app.py", line 1796, in dispatch_request
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 467, in wrapper
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     resp = resource(*args, **kwargs)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask/views.py", line 107, in view
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/flask_restful/__init__.py", line 582, in dispatch_request
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     resp = meth(*args, **kwargs)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/projects.py", line 151, in get
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self._get_project(project_id)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/api/projects.py", line 107, in _get_project
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     project = PROVIDERS.resource_api.get_project(project_id)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/common/manager.py", line 115, in wrapped
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     __ret_val = __f(*args, **kwargs)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/decorator.py", line 232, in fun
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return caller(func, *(extras + args), **kw)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self.get_or_create(
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 1042, in get_or_create
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     with Lock(
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 185, in __enter__
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self._enter()
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 94, in _enter
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     generated = self._enter_create(value, createdtime)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/lock.py", line 178, in _enter_create
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self.creator()
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/dogpile/cache/region.py", line 995, in gen_value
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     created_value = creator(
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/resource/core.py", line 903, in get_project
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self.driver.get_project(project_id)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/resource/backends/sql.py", line 49, in get_project
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     return self._get_project(session, project_id).to_dict()
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application   File "/var/lib/kolla/venv/lib/python3.10/site-packages/keystone/resource/backends/sql.py", line 44, in _get_project
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application     raise exception.ProjectNotFound(project_id=project_id)
  2024-04-18 09:07:38.629 1014 ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: foo.

  
  We have traced it to a specific openstacksdk call and the issue can be reliably replicated (replace foo by a valid/existing project) by sourcing relevant admin credentials and running:
  python -c 'import openstack; conn = openstack.connect(); valid_project = conn.identity.find_project("foo"); print(f"Project id is {valid_project.id}")'

  The are no errors returned by the find_project call and correct
  projectID is displayed, but keystone logs the above error.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/2062247/+subscriptions