yahoo-eng-team team mailing list archive

[OpenStack Infra <2075349@xxxxxxxxxxxxxxxxxx>]

Reviewed:  https://review.opendev.org/c/openstack/keystone/+/925553
Committed: https://opendev.org/openstack/keystone/commit/7ac0c3cd33214ff3c926e2b5316b637892d701fb
Submitter: "Zuul (22348)"
Branch:    master

commit 7ac0c3cd33214ff3c926e2b5316b637892d701fb
Author: Jadon Naas <jadon.naas@xxxxxxxxxxxxx>
Date:   Thu Aug 1 21:10:43 2024 -0400

    Update OIDC Apache config to avoid masking Keystone API endpoint
    
    The current configuration for the OIDCRedirectURI results in
    mod_auth_openidc masking the Keystone federation authentication
    endpoint, which results in incorrect responses to requests for
    Keystone tokens. This change updates the documentation to
    recommend using a vanity URL that does not match a Keystone
    API endpoint.
    
    Closes-Bug: 2075349
    Change-Id: I1dfba5c71da68522fdb6059f0dc03cddc74cb07d


** Changed in: keystone
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/2075349

Title:
  JSONDecodeError when OIDCRedirectURI is the same as the Keystone OIDC
  auth endpoint

Status in OpenStack Keystone OIDC Integration Charm:
  In Progress
Status in OpenStack Identity (keystone):
  Fix Released

Bug description:
  This bug is about test failures for jammy-caracal, jammy-bobcat, and
  jammy-antelope in cherry-pick commits from this change:

  https://review.opendev.org/c/openstack/charm-keystone-openidc/+/922049

  That change fixed some bugs in the Keystone OpenIDC charm and added
  some additional configuration options to help with proxies.

  The tests all fail with a JSONDecodeError during the Zaza tests for
  the Keystone OpenIDC charm. Here is an example of the error:

  Expecting value: line 1 column 1 (char 0)
  Traceback (most recent call last):
    File "/home/jadon/py3-venv/lib/python3.10/site-packages/requests/models.py", line 974, in json
      return complexjson.loads(self.text, **kwargs)
    File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
      return _default_decoder.decode(s)
    File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
      obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
      raise JSONDecodeError("Expecting value", s, err.value) from None
  json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/home/jadon/py3-venv/lib/python3.10/site-packages/cliff/app.py", line 414, in run_subcommand
      self.prepare_to_run_command(cmd)
    File "/home/jadon/py3-venv/lib/python3.10/site-packages/osc_lib/shell.py", line 516, in prepare_to_run_command
      self.client_manager.auth_ref
    File "/home/jadon/py3-venv/lib/python3.10/site-packages/osc_lib/clientmanager.py", line 208, in auth_ref
      self._auth_ref = self.auth.get_auth_ref(self.session)
    File "/home/jadon/py3-venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/federation.py", line 62, in get_auth_ref
      auth_ref = self.get_unscoped_auth_ref(session)
    File "/home/jadon/py3-venv/lib/python3.10/site-packages/keystoneauth1/identity/v3/oidc.py", line 293, in get_unscoped_auth_ref
      return access.create(resp=response)
    File "/home/jadon/py3-venv/lib/python3.10/site-packages/keystoneauth1/access/access.py", line 36, in create
      body = resp.json()
    File "/home/jadon/py3-venv/lib/python3.10/site-packages/requests/models.py", line 978, in json
      raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
  requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
  clean_up ListServer: Expecting value: line 1 column 1 (char 0)
  END return value: 1

  According to debug output, the failure happens during the OIDC
  authentication flow. Testing using the OpenStack CLI shows the failure
  happen right after this request:

  REQ: curl -g -i --insecure -X POST https://10.70.143.111:5000/v3/OS-FEDERATION/identity_providers/keycloak/protocols/openid/auth -H "Authorization: {SHA256}45dbb29ea555e0bd24995cbb1481c8ac66c2d03383bc0c335be977d0daaf6959" -H "User-Agent: openstacksdk/3.3.0 keystoneauth1/5.7.0 python-requests/2.32.3 CPython/3.10.12"
  Starting new HTTPS connection (1): 10.70.143.111:5000
  RESP: [200] Connection: Keep-Alive Content-Length: 0 Date: Tue, 30 Jul 2024 19:28:17 GMT Keep-Alive: timeout=75, max=1000 Server: Apache/2.4.52 (Ubuntu)
  RESP BODY: Omitted, Content-Type is set to None. Only text/plain, application/json responses have their bodies logged.

  This request is unusual in that the request is a POST request with no
  request body, and the response is an empty response. The empty
  response causes the JSONDecodeError because the keystoneauth package
  expects a JSON document to return from the request for a Keystone
  token. The empty response causes the JSONDecodeError because an empty
  string is not a valid document.

  This strange behavior happens due to a misconfiguration in the
  mod_auth_openidc Apache configuration. I looked up how Kolla-Ansible
  configures OpenIDC in Keystone, and I noticed that they used a
  different value for the OIDCRedirectURI in the mod_auth_openidc Apache
  configuration than the Keystone OpenIDC charm. The value of
  OIDCRedirectURI is supposed to be a fake URI that does not map to any
  real URI in the protected service. The fake URI should be protected by
  mod_auth_openidc in Apache's configuration. When someone accesses the
  path in OIDCRedirectURI mod_auth_openidc handles the request and
  returns an empty response.

  In our configuration, the OIDCRedirectURI is the same URI as the
  Keystone federation protocol authentication endpoint. Keystone
  generates the federation protocol authentication endpoint dynamically
  based on the identity provider and federation protocol configured in
  Keystone. Here is an example of a federation protocol endpoint for a
  Keystone identity provider named "keycloak" using a Keystone
  federation protocol called "openid":

  /v3/OS-FEDERATION/identity_providers/keycloak/protocols/openid/auth

  When Keystone is attempting to access the Keystone federation protocol
  authentication endpoint, mod_auth_openidc is intercepting the request
  and returning an empty response. This is blocking Keystone from
  properly handling the OIDC authentication flow.

  If you change OIDCRedirectURI to something other than your federation
  protocol authentication endpoint, Keystone can complete the OIDC
  authentication flow successfully.

  This contradicts the configuration outlined in the official Keystone
  federation configuration documentation at
  https://docs.openstack.org/keystone/latest/admin/federation/configure_federation.html#configure-
  mod-auth-openidc.

To manage notifications about this bug go to:
https://bugs.launchpad.net/charm-keystone-openidc/+bug/2075349/+subscriptions