← Back to team overview

yahoo-eng-team team mailing list archive

  1. yahoo-eng-team team
  2. Mailing list archive
  3. Message #83583

[Bug 1891244] Re: *-grenade-multinode jobs fail with UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte

  Thread Previous • Date Previous • • Thread Next 
Reviewed:  https://review.opendev.org/745752
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=7d6c71ba26694c21110280e741b9ffe2d36a94ca
Submitter: Zuul
Branch:    master

commit 7d6c71ba26694c21110280e741b9ffe2d36a94ca
Author: melanie witt <melwittt@xxxxxxxxx>
Date:   Tue Aug 11 21:19:01 2020 +0000

    Support format for msgpack < 1.0 in token formatter
    
    msgpack v1.0 changed its data format [1] and during a rolling upgrade, attempts
    to unpack cached tokens with old data format with the new default raw=False
    result in the following error:
    
      UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in
      position 3: invalid start byte
    
    This passes raw=True to support backward-compat with the old format
    until we are guaranteed to have msgpack >= 1.0 in the N-1 release of
    a rolling upgrade.
    
    Closes-Bug: #1891244
    
    [1]
    https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10
    
    Change-Id: I6c61df6c097fef698c659c79402c4381ec7f3586


** Changed in: keystone
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1891244

Title:
  *-grenade-multinode jobs fail with UnicodeDecodeError: 'utf-8' codec
  can't decode byte 0x87 in position 3: invalid start byte

Status in OpenStack Identity (keystone):
  Fix Released

Bug description:
  With a recent bump of the upper constraint for the msgpack library to
  version v1.0 [1], the *-grenade-multinode gate jobs fail with the
  following trace.

  The data format was changed in msgpack v1.0 [2], so during a rolling
  upgrade, tokens cached from the N-1 release (msgpack < v1.0) will fail
  to unpack with the new format in msgpack v1.0.

  Aug 11 14:45:47.321414 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239fb20e-9ca7-476c-ba19-4c6707192097 None None] Validating token access rules against request {{(pid=29023) validate_allowed_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:545}}
  Aug 11 14:45:47.325427 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] RBAC: auth_context: {'token': <TokenModel (audit_id=XBmEG0BZRNCTDw5-ANYeHw, audit_chain_id=['XBmEG0BZRNCTDw5-ANYeHw']) at 0x7f6d17d4d0f0>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60203c873e874a11ae4e8ee43f6ed7cc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cdb7b38e6fc84c20b1aa05d012d599f7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=29023) fill_context /opt/stack/new/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}}
  Aug 11 14:45:47.325958 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] REQUEST_METHOD: `GET` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:27}}
  Aug 11 14:45:47.326214 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] SCRIPT_NAME: `/identity` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:28}}
  Aug 11 14:45:47.326462 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:29}}
  Aug 11 14:45:47.327842 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: DEBUG keystone.common.fernet_utils [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=29023) load_keys /opt/stack/new/keystone/keystone/common/fernet_utils.py:289}}
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: CRITICAL keystone [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] Unhandled error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone Traceback (most recent call last):
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2464, in __call__
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.wsgi_app(environ, start_response)
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.app(environ, start_response)
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/base.py", line 124, in __call__
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     response = req.get_response(self.application)
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     application, catch_exc_info=False)
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 143, in __call__
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return resp(environ, start_response)
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
  Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/base.py", line 124, in __call__
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     response = req.get_response(self.application)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     application, catch_exc_info=False)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/osprofiler/web.py", line 112, in __call__
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return request.get_response(self.application)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     application, catch_exc_info=False)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     response = req.get_response(self.application)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     application, catch_exc_info=False)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/opt/stack/new/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.app(environ, start_response)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     resp = self.call_func(req, *args, **kw)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.func(req, *args, **kwargs)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     response = req.get_response(self._app)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     application, catch_exc_info=False)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     app_iter = application(self.environ, start_response)
  Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return app(environ, start_response)
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2450, in wsgi_app
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     response = self.handle_exception(e)
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return original_handler(e)
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return original_handler(e)
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return original_handler(e)
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   [Previous line repeated 27 more times]
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1867, in handle_exception
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     reraise(exc_type, exc_value, tb)
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 38, in reraise
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     raise value.with_traceback(tb)
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     response = self.full_dispatch_request()
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     rv = self.handle_user_exception(e)
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return original_handler(e)
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
  Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return original_handler(e)
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return original_handler(e)
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   [Previous line repeated 27 more times]
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     reraise(exc_type, exc_value, tb)
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 38, in reraise
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     raise value.with_traceback(tb)
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     rv = self.dispatch_request()
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.view_functions[rule.endpoint](**req.view_args)
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 468, in wrapper
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     resp = resource(*args, **kwargs)
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask/views.py", line 89, in view
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.dispatch_request(*args, **kwargs)
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 583, in dispatch_request
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     resp = meth(*args, **kwargs)
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/opt/stack/new/keystone/keystone/api/auth.py", line 285, in get
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     ENFORCER.enforce_call(action='identity:validate_token')
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/opt/stack/new/keystone/keystone/common/rbac_enforcer/enforcer.py", line 414, in enforce_call
  Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     subj_token_target_data = cls._extract_subject_token_target_data()
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/opt/stack/new/keystone/keystone/common/rbac_enforcer/enforcer.py", line 256, in _extract_subject_token_target_data
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     access_rules_support=access_rules_support
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/opt/stack/new/keystone/keystone/common/manager.py", line 115, in wrapped
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     __ret_val = __f(*args, **kwargs)
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/opt/stack/new/keystone/keystone/token/provider.py", line 145, in validate_token
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     token = self._validate_token(token_id)
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "<decorator-gen-26>", line 2, in _validate_token
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 1357, in get_or_create_for_user_func
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     key, user_func, timeout, should_cache_fn, (arg, kw)
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 959, in get_or_create
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     async_creator,
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 185, in __enter__
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self._enter()
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 94, in _enter
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     generated = self._enter_create(value, createdtime)
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 178, in _enter_create
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.creator()
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 913, in gen_value
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     *creator_args[0], **creator_args[1]
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/opt/stack/new/keystone/keystone/token/provider.py", line 158, in _validate_token
  Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     expires_at) = self.driver.validate_token(token_id)
  Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/opt/stack/new/keystone/keystone/token/providers/fernet/core.py", line 96, in validate_token
  Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     return self.token_formatter.validate_token(token_id)
  Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "/opt/stack/new/keystone/keystone/token/token_formatters.py", line 173, in validate_token
  Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone     versioned_payload = msgpack.unpackb(serialized_payload)
  Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone   File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb
  Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte
  Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 devstack@keystone.service[29022]: ERROR keystone

  [1] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373
  [2] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1891244/+subscriptions

References

  Thread Previous • Date Previous • • Thread Next