yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1613901] Re: String "..%c0%af" causes 500 errors in multiple locations in Keystone v3

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Charles Neill <charles.neill@xxxxxxxxxxxxx>
Date: Wed, 28 Sep 2016 00:22:27 -0000
Reply-to: Bug 1613901 <1613901@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
This affects a number of other OpenStack projects in a similar way,
including:

- Neutron
- Cinder
- Glance

More projects may be affected that we are unaware of.

=====================================================
Example traceback from Neutron
=====================================================

2016-09-28 00:16:43.342 1218 DEBUG neutron.wsgi [-] (1218) accepted ('10.0.2.2', 50029) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/__init__.py", line 851, in emit
    msg = self.format(record)
  File "/usr/local/lib/python2.7/dist-packages/oslo_log/handlers.py", line 76, in format
    return logging.StreamHandler.format(self, record)
  File "/usr/lib/python2.7/logging/__init__.py", line 724, in format
    return fmt.format(record)
  File "/usr/local/lib/python2.7/dist-packages/oslo_log/formatters.py", line 297, in format
    return logging.Formatter.format(self, record)
  File "/usr/lib/python2.7/logging/__init__.py", line 464, in format
    record.message = record.getMessage()
  File "/usr/lib/python2.7/logging/__init__.py", line 328, in getMessage
    msg = msg % self.args
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1164, in as_text
    bytes = self.as_bytes()
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1135, in as_bytes
    url = self.url
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
    url = self.path_url
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
    bpath_info = bytes_(self.path_info, self.url_encoding)
  File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
    return req.encget(key, encattr=encattr)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
    return val.decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 11: invalid start byte
Logged from file catch_errors.py, line 41
2016-09-28 00:16:43.489 1218 INFO neutron.wsgi [req-37cfe540-c134-4ec0-91d2-56687f38ffd5 admin -] 10.0.2.2 - - [28/Sep/2016 00:16:43] "PUT /v2.0/flavors/..%c0%af HTTP/1.1" 500 414 0.140984

=====================================================
Example traceback from Cinder
=====================================================

2016-09-27 23:50:00.142 5986 DEBUG eventlet.wsgi.server [-] (5986) accepted ('10.0.2.2', 49862) server /usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py:868
2016-09-27 23:50:00.198 ERROR cinder.api.middleware.fault [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Caught error: <type 'exceptions.UnicodeDecodeError'> 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault Traceback (most recent call last):
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 79, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return req.get_response(self.application)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 323, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     response = req.get_response(self._app)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     application, catch_exc_info=False)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     app_iter = application(self.environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/routes/middleware.py", line 141, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     response = self.app(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return resp(environ, start_response)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     resp = self.call_func(req, *args, **self.kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return self.func(req, *args, **kwargs)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/opt/stack/cinder/cinder/api/openstack/wsgi.py", line 817, in __call__
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     "url": request.url})
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     url = self.path_url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     bpath_info = bytes_(self.path_info, self.url_encoding)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return req.encget(key, encattr=encattr)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return val.decode(encoding)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault   File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault     return codecs.utf_8_decode(input, errors, True)
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte
2016-09-27 23:50:00.198 5986 ERROR cinder.api.middleware.fault
2016-09-27 23:50:00.224 INFO eventlet.wsgi.server [req-7ec3610c-9cdc-4f7d-b69c-36c385d0fa10 admin] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/paste/urlmap.py", line 216, in __call__
    return app(environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/request_id.py", line 37, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 81, in __call__
    return self._error(ex, req)
  File "/opt/stack/cinder/cinder/api/middleware/fault.py", line 56, in _error
    msg_dict = dict(url=req.url, status=status)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 504, in url
    url = self.path_url
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 476, in path_url
    bpath_info = bytes_(self.path_info, self.url_encoding)
  File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
    return req.encget(key, encattr=encattr)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
    return val.decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 3: invalid start byte

2016-09-27 23:50:00.228 INFO eventlet.wsgi.server [req-7ec3610c-9cdc-
4f7d-b69c-36c385d0fa10 admin] 10.0.2.2 "POST /v2/..%c0%af/backups
HTTP/1.1" status: 500  len: 139 time: 0.0827849


=====================================================
Example traceback from Glance
=====================================================

2016-09-28 00:11:55.844 32495 INFO eventlet.wsgi.server [-] Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 481, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 126, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1299, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1263, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/base.py", line 123, in __call__
    response = self.process_request(req)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 148, in __call__
    return self.func(req, *args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/oslo_middleware/healthcheck/__init__.py", line 361, in process_request
    if req.path != self._path:
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 485, in path
    bpath = bytes_(self.path_info, self.url_encoding)
  File "/usr/local/lib/python2.7/dist-packages/webob/descriptors.py", line 68, in fget
    return req.encget(key, encattr=encattr)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 177, in encget
    return val.decode(encoding)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 13: invalid start byte

2016-09-28 00:11:55.854 32495 INFO eventlet.wsgi.server [-] 10.0.2.2 - -
[28/Sep/2016 00:11:55] "GET /v2/images/..%c0%af/tags/..%c0%af HTTP/1.1"
500 139 0.043949


I'm not sure whether right approach here is to file a bug with webob (looks like there are several that haven't been resolved yet [1] [2]), or to fix this in each project's respective wsgi error-handling code.

[1] https://github.com/Pylons/webob/issues/115
[2] https://github.com/Pylons/webob/issues/161

** Bug watch added: github.com/Pylons/webob/issues #115
   https://github.com/Pylons/webob/issues/115

** Bug watch added: github.com/Pylons/webob/issues #161
   https://github.com/Pylons/webob/issues/161

** Summary changed:

- String "..%c0%af" causes 500 errors in multiple locations in Keystone v3
+ String "..%c0%af" causes 500 errors in multiple locations

** Also affects: neutron
   Importance: Undecided
       Status: New

** Also affects: glance
   Importance: Undecided
       Status: New

** Also affects: cinder
   Importance: Undecided
       Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1613901

Title:
  String "..%c0%af" causes 500 errors in multiple locations

Status in Cinder:
  New
Status in Glance:
  New
Status in OpenStack Identity (keystone):
  Confirmed
Status in neutron:
  New
Status in OpenStack Security Advisory:
  Won't Fix

Bug description:

  While doing some testing on Keystone using Syntribos
  (https://github.com/openstack/syntribos), our team (myself, Michael
  Dong, Rahul U Nair, Vinay Potluri, Aastha Dixit, and Khanak Nangia)
  noticed that we got 500 status codes when the string "..%c0%af" was
  inserted in various places in the URL for different types of requests.

  Here are some examples:

  =========

  DELETE /v3/policies/..%c0%af HTTP/1.1
  Host: [REDACTED]:5000
  Connection: close
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-requests/2.11.0
  X-Auth-Token: [REDACTED]
  Content-Length: 0

  HTTP/1.1 500 Internal Server Error
  Date: Tue, 16 Aug 2016 22:04:27 GMT
  Server: Apache/2.4.7 (Ubuntu)
  Vary: X-Auth-Token
  X-Distribution: Ubuntu
  x-openstack-request-id: req-238fd5a9-be45-41f2-893a-97b513b27af3
  Content-Length: 143
  Connection: close
  Content-Type: application/json

  {"error": {"message": "An unexpected error prevented the server from
  fulfilling your request.", "code": 500, "title": "Internal Server
  Error"}}

  =========

  PATCH /v3/policies/..%c0%af HTTP/1.1
  Host: [REDACTED]:5000
  Connection: close
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-requests/2.11.0
  Content-type: application/json
  X-Auth-Token: [REDACTED]
  Content-Length: 70

  {"type": "--serialization-mime-type--", "blob": "--serialized-blob--"}

  HTTP/1.1 500 Internal Server Error
  Date: Tue, 16 Aug 2016 22:05:36 GMT
  Server: Apache/2.4.7 (Ubuntu)
  Vary: X-Auth-Token
  X-Distribution: Ubuntu
  x-openstack-request-id: req-57a41600-02b4-4d2a-b3e9-40f7724d65f2
  Content-Length: 143
  Connection: close
  Content-Type: application/json

  {"error": {"message": "An unexpected error prevented the server from
  fulfilling your request.", "code": 500, "title": "Internal Server
  Error"}}

  =========

  GET /v3/domains/0426ac1e48f642ef9544c2251e07e261/groups/..%c0%af/roles HTTP/1.1
  Host: [REDACTED]:5000
  Connection: close
  Accept-Encoding: gzip, deflate
  Accept: application/json
  User-Agent: python-requests/2.11.0
  X-Auth-Token: [REDACTED]

  HTTP/1.1 500 Internal Server Error
  Date: Tue, 16 Aug 2016 22:07:09 GMT
  Server: Apache/2.4.7 (Ubuntu)
  Vary: X-Auth-Token
  X-Distribution: Ubuntu
  x-openstack-request-id: req-02313f77-63c6-4aa8-a87e-e3d2a13ad6b7
  Content-Length: 143
  Connection: close
  Content-Type: application/json

  {"error": {"message": "An unexpected error prevented the server from
  fulfilling your request.", "code": 500, "title": "Internal Server
  Error"}}

  =========

  I've marked this as a security issue as a precaution in case it turns
  out that there is a more serious vulnerability underlying these
  errors. We have no reason to suspect that there is a greater
  vulnerability at this time, but given the many endpoints this seems to
  affect, I figured caution was worthwhile since this may be a
  framework-wide issue. Feel free to make this public if it is
  determined not to be security-impacting.

  Here is a (possibly incomplete) list of affected endpoints. Inserting
  the string "..%c0%af" in any or all of the spots labeled "HERE" should
  yield a 500 error. As you can see, virtually all v3 endpoints exhibit
  this behavior.

  =========

  [GET|PATCH|DELETE] /v3/endpoints/[HERE]

  [GET|PATCH]       /v3/domains/[HERE]
  GET               /v3/domains/[HERE]/groups/[HERE]/roles
  [HEAD|PUT|DELETE] /v3/domains/[HERE]/groups/[HERE]/roles/[HERE]
  GET               /v3/domains/[HERE]/users/[HERE]/roles
  [HEAD|DELETE]     /v3/domains/[HERE]/users/[HERE]/roles/[HERE]

  [GET|PATCH|DELETE] /v3/groups/[HERE]
  [HEAD|PUT|DELETE]  /v3/groups[HERE]/users/[HERE]

  [POST|DELETE] /v3/keys/[HERE]

  [GET|PATCH|DELETE] /v3/policies/[HERE]
  [GET|PUT|DELETE]   /v3/policies/[HERE]/OS-ENDPOINT-POLICY/endpoints/[HERE]
  [GET|HEAD]         /v3/policies/[HERE]/OS-ENDPOINT-POLICY/policy
  [GET|PUT|DELETE]   /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/[HERE]
  [PUT|DELETE]       /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/[HERE]
  [GET|PUT|DELETE]   /v3/policies/[HERE]/OS-ENDPOINT-POLICY/services/regions/[HERE]

  [GET|PATCH|DELETE] /v3/projects/[HERE]
  [DELETE|PATCH]     /v3/projects/[HERE]/cascade
  GET                /v3/projects/[HERE]/groups/[HERE]/roles
  GET                /v3/projects/[HERE]/users/[HERE]/roles
  [HEAD|PUT|DELETE]  /v3/projects/[HERE]/groups/[HERE]/roles/[HERE]

  [GET|PATCH|DELETE] /v3/regions/[HERE]

  [PATCH|DELETE] /v3/roles/[HERE]

  [GET|PATCH|DELETE] /v3/services/[HERE]

  [GET|PATCH|DELETE] /v3/users/[HERE]
  GET /v3/users/[HERE]/groups
  POST /v3/users/[HERE]/password
  GET /v3/users/[HERE]/projects

  GET                /v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]/roles/[HERE]
  [GET|PATCH|DELETE] /v3/OS-OAUTH1/consumers/[HERE]
  [GET|DELETE]       /v3/OS-OAUTH1/users/[HERE]/access_tokens/[HERE]

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1613901/+subscriptions