yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1595468] [NEW] Cannot encode revokeTree object when mongo is configured as cache

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: Jose Castro Leon <jose.castro.leon@xxxxxxx>
Date: Thu, 23 Jun 2016 10:05:05 -0000
Reply-to: Bug 1595468 <1595468@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

When you configure mongo as the cache subsystem for keystone, it fails
to store the revoke tree in the cache. This is due that the
basetransform only expects single types or dictionaries.

    def transform_incoming(self, son, collection):
        """Used while saving data to MongoDB."""
        for (key, value) in list(son.items()):
            if isinstance(value, api.CachedValue):
                son[key] = value.payload  # key is 'value' field here
                son['meta'] = value.metadata
            elif isinstance(value, dict):  # Make sure we recurse into sub-docs
                son[key] = self.transform_incoming(value, collection)
        return son

In mitaka this code has been sent to oslo_cache.mongo but the issue is
still there

As a result there is a 500 error on the response for validating a token.

This is the exception I am getting

2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi [req-9508d2e4-4e4c-4fd1-8daa-bc7c3456a320 - - - - -] Cannot encode object: <keystone.contrib.revoke.model.RevokeTree object at 0x7f5b0f177a90>
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi Traceback (most recent call last):
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 452, in __call__
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     response = self.process_request(request)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/middleware/core.py", line 305, in process_request
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     auth_context = self._build_auth_context(request)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/middleware/core.py", line 193, in _build_auth_context
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     token_data=self.token_provider_api.validate_token(token_id))
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 190, in validate_token
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self._is_valid_token(token)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 296, in _is_valid_token
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.check_revocation(token)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 230, in check_revocation
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     return self.check_revocation_v3(token)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 223, in check_revocation_v3
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.revoke_api.check_token(token_values)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/contrib/revoke/core.py", line 226, in check_token
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     if self._get_revoke_tree().is_revoked(token_values):
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1040, in decorate
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     should_cache_fn)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 651, in get_or_create
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     async_creator) as value:
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 158, in __enter__
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     return self._enter()
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 98, in _enter
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     generated = self._enter_create(createdtime)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 149, in _enter_create
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     created = self.creator()
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 624, in gen_value
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.backend.set(key, value)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/cache/backends/mongo.py", line 186, in set
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.client.set(key, value)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/cache/backends/mongo.py", line 443, in set
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     **self.meth_kwargs)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1390, in find_and_modify
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     **kwargs)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 388, in command
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     result = self["$cmd"].find_one(command, **extra_opts)
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 596, in find_one
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     for result in self.find(spec_or_id, *args, **kwargs).limit(-1):
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 814, in next
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     if len(self.__data) or self._refresh():
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 763, in _refresh
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.__uuid_subtype))
2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi InvalidDocument: Cannot encode object: <keystone.contrib.revoke.model.RevokeTree object at 0x7f5b0f177a90>


I am opening this here, because with the change from revoketree to revokeevents, maybe there is something to be checked here as well.

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1595468

Title:
  Cannot encode revokeTree object when mongo is configured as cache

Status in OpenStack Identity (keystone):
  New

Bug description:
  When you configure mongo as the cache subsystem for keystone, it fails
  to store the revoke tree in the cache. This is due that the
  basetransform only expects single types or dictionaries.

      def transform_incoming(self, son, collection):
          """Used while saving data to MongoDB."""
          for (key, value) in list(son.items()):
              if isinstance(value, api.CachedValue):
                  son[key] = value.payload  # key is 'value' field here
                  son['meta'] = value.metadata
              elif isinstance(value, dict):  # Make sure we recurse into sub-docs
                  son[key] = self.transform_incoming(value, collection)
          return son

  In mitaka this code has been sent to oslo_cache.mongo but the issue is
  still there

  As a result there is a 500 error on the response for validating a
  token.

  This is the exception I am getting

  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi [req-9508d2e4-4e4c-4fd1-8daa-bc7c3456a320 - - - - -] Cannot encode object: <keystone.contrib.revoke.model.RevokeTree object at 0x7f5b0f177a90>
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi Traceback (most recent call last):
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 452, in __call__
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     response = self.process_request(request)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/middleware/core.py", line 305, in process_request
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     auth_context = self._build_auth_context(request)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/middleware/core.py", line 193, in _build_auth_context
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     token_data=self.token_provider_api.validate_token(token_id))
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 190, in validate_token
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self._is_valid_token(token)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 296, in _is_valid_token
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.check_revocation(token)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 230, in check_revocation
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     return self.check_revocation_v3(token)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/provider.py", line 223, in check_revocation_v3
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.revoke_api.check_token(token_values)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/contrib/revoke/core.py", line 226, in check_token
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     if self._get_revoke_tree().is_revoked(token_values):
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 1040, in decorate
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     should_cache_fn)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 651, in get_or_create
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     async_creator) as value:
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 158, in __enter__
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     return self._enter()
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 98, in _enter
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     generated = self._enter_create(createdtime)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 149, in _enter_create
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     created = self.creator()
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/dogpile/cache/region.py", line 624, in gen_value
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.backend.set(key, value)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/cache/backends/mongo.py", line 186, in set
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.client.set(key, value)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/cache/backends/mongo.py", line 443, in set
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     **self.meth_kwargs)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1390, in find_and_modify
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     **kwargs)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/database.py", line 388, in command
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     result = self["$cmd"].find_one(command, **extra_opts)
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 596, in find_one
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     for result in self.find(spec_or_id, *args, **kwargs).limit(-1):
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 814, in next
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     if len(self.__data) or self._refresh():
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/pymongo/cursor.py", line 763, in _refresh
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi     self.__uuid_subtype))
  2016-06-23 06:42:13.248 30288 ERROR keystone.common.wsgi InvalidDocument: Cannot encode object: <keystone.contrib.revoke.model.RevokeTree object at 0x7f5b0f177a90>

  
  I am opening this here, because with the change from revoketree to revokeevents, maybe there is something to be checked here as well.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1595468/+subscriptions
Follow ups

[Bug 1595468] Re: Cannot encode revokeTree object when mongo is configured as cache
From: Dolph Mathews, 2016-07-07