yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1988168] Re: Broken host:port splitting

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: OpenStack Infra <1988168@xxxxxxxxxxxxxxxxxx>
Date: Wed, 07 Sep 2022 16:04:18 -0000
Reply-to: Bug 1988168 <1988168@xxxxxxxxxxxxxxxxxx>
Sender: noreply@xxxxxxxxxxxxx
Reviewed:  https://review.opendev.org/c/openstack/keystone/+/855198
Committed: https://opendev.org/openstack/keystone/commit/6c35b366e3c8c6d7f47471b93f5315582301c5ef
Submitter: "Zuul (22348)"
Branch:    master

commit 6c35b366e3c8c6d7f47471b93f5315582301c5ef
Author: Bence Romsics <bence.romsics@xxxxxxxxx>
Date:   Mon Aug 29 16:03:44 2022 +0200

    Fix host:port handling
    
    When we check the EC2 signature without the port part of the host value
    received, we should properly split host:port. Keep in mind the splitting
    should work for values like [fc00::]:123 too.
    
    Change-Id: I1d90dfcea3568e2a9b22069daa428ea6a2a38bd6
    Closes-Bug: #1988168


** Changed in: keystone
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1988168

Title:
  Broken host:port splitting

Status in OpenStack Identity (keystone):
  Fix Released

Bug description:
  Our users found a bug while POSTing to /v3/ec2tokens. I could simplify
  the reproduction to this script:

  $ cat keystone-post-ec2tokens.sh 
  #! /bin/sh

  # source openrc admin admin
  # keystone-post-ec2tokens.sh http://127.0.0.1/identity/v3

  keystone_base_url="${1:?}"

  cleanup () {
      openstack ec2 credential delete "$access"
  }
  trap cleanup EXIT

  #host="localhost"
  host="localhost:123"
  #host="1.2.3.4:123"
  #host="[fc00::]:123"
  access="$( openstack ec2 credential create -f value -c access )"
  secret="$( openstack ec2 credential show "$access" -f value -c secret )"
  signature="intentionally-invalid"

  cat <<EOF |
  {
      "credentials": {
          "access": "$access",
          "host": "$host",
          "params": {
              "Action": "Test",
              "SignatureMethod": "HmacSHA256",
              "SignatureVersion": "2",
              "Timestamp": "2000-01-01T00:00:00Z"
          },
          "path": "/",
          "secret": "$secret",
          "signature": "$signature",
          "verb": "GET"
      }
  }
  EOF
  curl \
      -s \
      -d @- \
      -H "Content-Type: application/json" \
      -H "Accept: application/json" \
      -X POST \
      "$keystone_base_url/ec2tokens"
  END-OF-SCRIPT

  Using any of the host values containing a port number, keystone throws
  an Internal Server Error:

  ~/keystone-post-ec2tokens.sh http://127.0.0.1/identity/v3
  <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
  <html><head>
  <title>500 Internal Server Error</title>
  </head><body>
  <h1>Internal Server Error</h1>
  <p>The server encountered an internal error or
  misconfiguration and was unable to complete
  your request.</p>
  <p>Please contact the server administrator at 
   webmaster@localhost to inform them of the time this error occurred,
   and the actions you performed just before this error.</p>
  <p>More information about this error may be available
  in the server error log.</p>
  <hr>
  <address>Apache/2.4.52 (Ubuntu) Server at 127.0.0.1 Port 80</address>
  </body></html>

  With the following stack trace in the logs:

  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone Traceback (most recent call last):
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2548, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.wsgi_app(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.app(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self.application)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 143, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return resp(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self.application)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/osprofiler/web.py", line 111, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return request.get_response(self.application)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self.application)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.app(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.func(req, *args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = req.get_response(self._app)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     status, headers, app_iter = self.call_application(
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     app_iter = application(self.environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/werkzeug/middleware/dispatcher.py", line 78, in __call__
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return app(environ, start_response)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2528, in wsgi_app
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = self.handle_exception(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   [Previous line repeated 28 more times]
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 2525, in wsgi_app
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     response = self.full_dispatch_request()
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1822, in full_dispatch_request
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     rv = self.handle_user_exception(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 271, in error_router
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return original_handler(e)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   [Previous line repeated 28 more times]
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1820, in full_dispatch_request
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     rv = self.dispatch_request()
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/app.py", line 1796, in dispatch_request
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 467, in wrapper
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = resource(*args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask/views.py", line 107, in view
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return current_app.ensure_sync(self.dispatch_request)(**kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/usr/local/lib/python3.10/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     resp = meth(*args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/server/flask/common.py", line 1064, in wrapper
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     return f(*args, **kwargs)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/api/ec2tokens.py", line 67, in post
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     token = self.handle_authenticate()
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/api/_shared/EC2_S3_Resource.py", line 122, in handle_authenticate
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     self._check_signature(cred_data, credentials)
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone   File "/opt/stack/keystone/keystone/api/ec2tokens.py", line 45, in _check_signature
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone     hostname, _port = credentials.split(':')
  aug 30 11:53:59 devstack0 devstack@keystone.service[31882]: ERROR keystone AttributeError: 'dict' object has no attribute 'split'

  Keystone raises on this line:

  https://opendev.org/openstack/keystone/src/commit/051aca8e8a488efc51817463dab8e4daafbbbf59/keystone/api/ec2tokens.py#L45

  Clearly the author wanted to split credentials['host'] and not
  credentials.

  Without the bug present, keystone should reject the request as
  unauthorized (since the signature is not computed properly).

  devstack 90e5479f
  keystone 051aca8e8

  Posting a proposed fix soon.

  By the way: I found the /v2.0 api-ref for /ec2tokens, which marked it
  as deprecated. Despite this I found the same resources working under
  /v3, but I could not find anything about them in the /v3 api-ref. Did
  I miss something?

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1988168/+subscriptions
References

[Bug 1988168] [NEW] Broken host:port splitting
From: Bence Romsics, 2022-08-30