yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1988168] [NEW] Broken host:port splitting

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Bence Romsics <1988168@xxxxxxxxxxxxxxxxxx>
Date: Tue, 30 Aug 2022 12:16:19 -0000
Reply-to: Bug 1988168 <1988168@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Public bug reported:

Our users found a bug while POSTing to /v3/ec2tokens. I could simplify
the reproduction to this script:

$ cat keystone-post-ec2tokens.sh 
#! /bin/sh

# source openrc admin admin
# keystone-post-ec2tokens.sh http://127.0.0.1/identity/v3

keystone_base_url="${1:?}"

cleanup () {
    openstack ec2 credential delete "$access"
}
trap cleanup EXIT

#host="localhost"
host="localhost:123"
#host="1.2.3.4:123"
#host="[fc00::]:123"
access="$( openstack ec2 credential create -f value -c access )"
secret="$( openstack ec2 credential show "$access" -f value -c secret )"
signature="intentionally-invalid"

cat <<EOF |
{
    "credentials": {
        "access": "$access",
        "host": "$host",
        "params": {
            "Action": "Test",
            "SignatureMethod": "HmacSHA256",
            "SignatureVersion": "2",
            "Timestamp": "2000-01-01T00:00:00Z"
        },
        "path": "/",
        "secret": "$secret",
        "signature": "$signature",
        "verb": "GET"
    }
}
EOF
curl \
    -s \
    -d @- \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -X POST \
    "$keystone_base_url/ec2tokens"
END-OF-SCRIPT

Using any of the host values containing a port number, keystone throws
an Internal Server Error:

~/keystone-post-ec2tokens.sh http://127.0.0.1/identity/v3
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at 
 webmaster@localhost to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
<hr>
<address>Apache/2.4.52 (Ubuntu) Server at 127.0.0.1 Port 80</address>
</body></html>

With the following stack trace in the logs:

aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone Traceback (most recent call last):
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2548, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.wsgi_app(environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.app(environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self.application)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 143, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return resp(environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self.application)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/osprofiler/web.py", line 111, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return request.get_response(self.application)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self.application)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.app(environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self._app)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/werkzeug/middleware/dispatcher.py", line 78, in __call__
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return app(environ, start_response)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2528, in wsgi_app
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = self.handle_exception(e)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   [Previous line repeated 28 more times]
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2525, in wsgi_app
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = self.full_dispatch_request()
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1822, in full_dispatch_request
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     rv = self.handle_user_exception(e)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   [Previous line repeated 28 more times]
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1820, in full_dispatch_request
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     rv = self.dispatch_request()
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1796, in dispatch_request
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 467, in wrapper
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = resource(*args, **kwargs)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/views.py", line 107, in view
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return current_app.ensure_sync(self.dispatch_request)(**kwargs)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = meth(*args, **kwargs)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/common.py", line 1064, in wrapper
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return f(*args, **kwargs)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/api/ec2tokens.py", line 67, in post
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     token = self.handle_authenticate()
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/api/_shared/EC2_S3_Resource.py", line 122, in handle_authenticate
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     self._check_signature(cred_data, credentials)
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/api/ec2tokens.py", line 45, in _check_signature
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     hostname, _port = credentials.split(':')
aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone AttributeError: 'dict' object has no attribute 'split'

Keystone raises on this line:

https://opendev.org/openstack/keystone/src/commit/051aca8e8a488efc51817463dab8e4daafbbbf59/keystone/api/ec2tokens.py#L45

Clearly the author wanted to split credentials['host'] and not
credentials.

Without the bug present, keystone should reject the request as
unauthorized (since the signature is not computed properly).

devstack 90e5479f
keystone 051aca8e8

Posting a proposed fix soon.

By the way: I found the /v2.0 api-ref for /ec2tokens, which marked it as
deprecated. Despite this I found the same resources working under /v3,
but I could not find anything about them in the /v3 api-ref. Did I miss
something?

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1988168

Title:
  Broken host:port splitting

Status in OpenStack Identity (keystone):
  New

Bug description:
  Our users found a bug while POSTing to /v3/ec2tokens. I could simplify
  the reproduction to this script:

  $ cat keystone-post-ec2tokens.sh 
  #! /bin/sh

  # source openrc admin admin
  # keystone-post-ec2tokens.sh http://127.0.0.1/identity/v3

  keystone_base_url="${1:?}"

  cleanup () {
      openstack ec2 credential delete "$access"
  }
  trap cleanup EXIT

  #host="localhost"
  host="localhost:123"
  #host="1.2.3.4:123"
  #host="[fc00::]:123"
  access="$( openstack ec2 credential create -f value -c access )"
  secret="$( openstack ec2 credential show "$access" -f value -c secret )"
  signature="intentionally-invalid"

  cat <<EOF |
  {
      "credentials": {
          "access": "$access",
          "host": "$host",
          "params": {
              "Action": "Test",
              "SignatureMethod": "HmacSHA256",
              "SignatureVersion": "2",
              "Timestamp": "2000-01-01T00:00:00Z"
          },
          "path": "/",
          "secret": "$secret",
          "signature": "$signature",
          "verb": "GET"
      }
  }
  EOF
  curl \
      -s \
      -d @- \
      -H "Content-Type: application/json" \
      -H "Accept: application/json" \
      -X POST \
      "$keystone_base_url/ec2tokens"
  END-OF-SCRIPT

  Using any of the host values containing a port number, keystone throws
  an Internal Server Error:

  ~/keystone-post-ec2tokens.sh http://127.0.0.1/identity/v3
  <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at 
   webmaster@localhost to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  <hr>
  <address>Apache/2.4.52 (Ubuntu) Server at 127.0.0.1 Port 80</address>
  </body></html>

  With the following stack trace in the logs:

  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone Traceback (most recent call last):
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2548, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.wsgi_app(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.app(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self.application)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 143, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return resp(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self.application)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/osprofiler/web.py", line 111, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return request.get_response(self.application)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self.application)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.app(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self._app)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/werkzeug/middleware/dispatcher.py", line 78, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return app(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2528, in wsgi_app
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = self.handle_exception(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   [Previous line repeated 28 more times]
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2525, in wsgi_app
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = self.full_dispatch_request()
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1822, in full_dispatch_request
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     rv = self.handle_user_exception(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   [Previous line repeated 28 more times]
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1820, in full_dispatch_request
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     rv = self.dispatch_request()
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1796, in dispatch_request
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 467, in wrapper
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = resource(*args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/views.py", line 107, in view
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = meth(*args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/common.py", line 1064, in wrapper
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return f(*args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/api/ec2tokens.py", line 67, in post
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     token = self.handle_authenticate()
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/api/_shared/EC2_S3_Resource.py", line 122, in handle_authenticate
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     self._check_signature(cred_data, credentials)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/api/ec2tokens.py", line 45, in _check_signature
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     hostname, _port = credentials.split(':')
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone AttributeError: 'dict' object has no attribute 'split'

  Keystone raises on this line:

  https://opendev.org/openstack/keystone/src/commit/051aca8e8a488efc51817463dab8e4daafbbbf59/keystone/api/ec2tokens.py#L45

  Clearly the author wanted to split credentials['host'] and not
  credentials.

  Without the bug present, keystone should reject the request as
  unauthorized (since the signature is not computed properly).

  devstack 90e5479f
  keystone 051aca8e8

  Posting a proposed fix soon.

  By the way: I found the /v2.0 api-ref for /ec2tokens, which marked it
  as deprecated. Despite this I found the same resources working under
  /v3, but I could not find anything about them in the /v3 api-ref. Did
  I miss something?

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1988168/+subscriptions
Follow ups

[Bug 1988168] Re: Broken host:port splitting
From: OpenStack Infra, 2022-09-07