yahoo-eng-team team mailing list archive

Thread
Date
[Bug 1618705] [NEW] keystone.cache.redis config arguments url conflict host and port

To: yahoo-eng-team@xxxxxxxxxxxxxxxxxxx
From: chuang <nashuiliang@xxxxxxxxx>
Date: Wed, 31 Aug 2016 05:49:30 -0000
Reply-to: Bug 1618705 <1618705@xxxxxxxxxxxxxxxxxx>
Sender: bounces@xxxxxxxxxxxxx
Public bug reported:

version: 8.1.3.dev1
branch: stable/liberty

in my keystone.conf
```
[cache]
config_prefix = cache.keystone
expiration_time = 600
backend = dogpile.cache.redis
backend_argument = host:127.0.0.1
backend_argument = port:6380
backend_argument = db:0
```

Using this keystone.conf is not working.
```
2016-08-31 13:46:56.552 10387 CRITICAL keystone [-] AttributeError: 'list' object has no attribute 'decode'
2016-08-31 13:46:56.552 10387 ERROR keystone Traceback (most recent call last):
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/bin/keystone-wsgi-admin", line 34, in <module>
2016-08-31 13:46:56.552 10387 ERROR keystone     server = wss.make_server('', args.port, initialize_admin_application())
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/wsgi.py", line 78, in initialize_admin_application
2016-08-31 13:46:56.552 10387 ERROR keystone     return initialize_application('admin')
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/wsgi.py", line 64, in initialize_application
2016-08-31 13:46:56.552 10387 ERROR keystone     startup_application_fn=loadapp)
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/common.py", line 49, in setup_backends
2016-08-31 13:46:56.552 10387 ERROR keystone     drivers = backends.load_backends()
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/backends.py", line 33, in load_backends
2016-08-31 13:46:56.552 10387 ERROR keystone     cache.configure_cache_region(cache.REGION)
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/common/cache/core.py", line 142, in configure_cache_region
2016-08-31 13:46:56.552 10387 ERROR keystone     '%s.' % CONF.cache.config_prefix)
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/region.py", line 552, in configure_from_config
2016-08-31 13:46:56.552 10387 ERROR keystone     "%swrap" % prefix, None),
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/region.py", line 417, in configure
2016-08-31 13:46:56.552 10387 ERROR keystone     _config_prefix
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/api.py", line 81, in from_config_dict
2016-08-31 13:46:56.552 10387 ERROR keystone     for key in config_dict
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/backends/redis.py", line 109, in __init__
2016-08-31 13:46:56.552 10387 ERROR keystone     self.client = self._create_client()
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/backends/redis.py", line 129, in _create_client
2016-08-31 13:46:56.552 10387 ERROR keystone     return redis.StrictRedis.from_url(**args)
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/redis/client.py", line 391, in from_url
2016-08-31 13:46:56.552 10387 ERROR keystone     connection_pool = ConnectionPool.from_url(url, db=db, **kwargs)
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/redis/connection.py", line 774, in from_url
2016-08-31 13:46:56.552 10387 ERROR keystone     url = urlparse(url)
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 294, in urlparse
2016-08-31 13:46:56.552 10387 ERROR keystone     url, scheme, _coerce_result = _coerce_args(url, scheme)
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 114, in _coerce_args
2016-08-31 13:46:56.552 10387 ERROR keystone     return _decode_args(args) + (_encode_result,)
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 98, in _decode_args
2016-08-31 13:46:56.552 10387 ERROR keystone     return tuple(x.decode(encoding, errors) if x else '' for x in args)
2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 98, in <genexpr>
2016-08-31 13:46:56.552 10387 ERROR keystone     return tuple(x.decode(encoding, errors) if x else '' for x in args)
2016-08-31 13:46:56.552 10387 ERROR keystone AttributeError: 'list' object has no attribute 'decode'
2016-08-31 13:46:56.552 10387 ERROR keystone
```

Therefore, I view the code, find the reason.

Although I have not configured the `url` of `backend_argument`, the code
sets the default `url` values `CONF.cache.memcache_servers`(file
`keystone/common/cache/core.py`, line 112-113).

Therefore, the variable of `CONF.cache.memcache_servers` default value
is `["localhost:11211"]`(file `keystone/common/config.py`, line
356-359).

So, when the code runs after `build_cache_conf` (file `keystone/common/cache/core.py`, line 139), the configuration of `cache` contain:
```
url = ["localhost:11211"]
host = "127.0.0.1"
port = 6380
db = 0
```

I use the cache backend `dogpile.cache.redis`, when the code is
configured the `url` argument and creates `redis_client`, will use the
function `StrictRedis.from_url` and `url` argument, and ignore the
argument `host`,`port` and `db`.

If I will use the cache backend `dogpile.cache.redis`, I must configure the `keystone.conf`
```
[cache]
config_prefix = cache.keystone
expiration_time = 600
backend = dogpile.cache.redis
backend_argument = url:redis://192.168.7.51:6380/0
```

** Affects: keystone
     Importance: Undecided
         Status: New

** Description changed:

  version: 8.1.3.dev1
  branch: stable/liberty
  
  in my keystone.conf
  ```
  [cache]
  config_prefix = cache.keystone
  expiration_time = 600
  backend = dogpile.cache.redis
  backend_argument = host:127.0.0.1
  backend_argument = port:6380
  backend_argument = db:0
  ```
  
  Using this keystone.conf is not working.
  ```
  2016-08-31 13:46:56.552 10387 CRITICAL keystone [-] AttributeError: 'list' object has no attribute 'decode'
  2016-08-31 13:46:56.552 10387 ERROR keystone Traceback (most recent call last):
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/bin/keystone-wsgi-admin", line 34, in <module>
  2016-08-31 13:46:56.552 10387 ERROR keystone     server = wss.make_server('', args.port, initialize_admin_application())
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/wsgi.py", line 78, in initialize_admin_application
  2016-08-31 13:46:56.552 10387 ERROR keystone     return initialize_application('admin')
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/wsgi.py", line 64, in initialize_application
  2016-08-31 13:46:56.552 10387 ERROR keystone     startup_application_fn=loadapp)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/common.py", line 49, in setup_backends
  2016-08-31 13:46:56.552 10387 ERROR keystone     drivers = backends.load_backends()
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/backends.py", line 33, in load_backends
  2016-08-31 13:46:56.552 10387 ERROR keystone     cache.configure_cache_region(cache.REGION)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/common/cache/core.py", line 142, in configure_cache_region
  2016-08-31 13:46:56.552 10387 ERROR keystone     '%s.' % CONF.cache.config_prefix)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/region.py", line 552, in configure_from_config
  2016-08-31 13:46:56.552 10387 ERROR keystone     "%swrap" % prefix, None),
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/region.py", line 417, in configure
  2016-08-31 13:46:56.552 10387 ERROR keystone     _config_prefix
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/api.py", line 81, in from_config_dict
  2016-08-31 13:46:56.552 10387 ERROR keystone     for key in config_dict
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/backends/redis.py", line 109, in __init__
  2016-08-31 13:46:56.552 10387 ERROR keystone     self.client = self._create_client()
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/backends/redis.py", line 129, in _create_client
  2016-08-31 13:46:56.552 10387 ERROR keystone     return redis.StrictRedis.from_url(**args)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/redis/client.py", line 391, in from_url
  2016-08-31 13:46:56.552 10387 ERROR keystone     connection_pool = ConnectionPool.from_url(url, db=db, **kwargs)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/redis/connection.py", line 774, in from_url
  2016-08-31 13:46:56.552 10387 ERROR keystone     url = urlparse(url)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 294, in urlparse
  2016-08-31 13:46:56.552 10387 ERROR keystone     url, scheme, _coerce_result = _coerce_args(url, scheme)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 114, in _coerce_args
  2016-08-31 13:46:56.552 10387 ERROR keystone     return _decode_args(args) + (_encode_result,)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 98, in _decode_args
  2016-08-31 13:46:56.552 10387 ERROR keystone     return tuple(x.decode(encoding, errors) if x else '' for x in args)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 98, in <genexpr>
  2016-08-31 13:46:56.552 10387 ERROR keystone     return tuple(x.decode(encoding, errors) if x else '' for x in args)
  2016-08-31 13:46:56.552 10387 ERROR keystone AttributeError: 'list' object has no attribute 'decode'
- 2016-08-31 13:46:56.552 10387 ERROR keystone 
+ 2016-08-31 13:46:56.552 10387 ERROR keystone
  ```
  
  Therefore, I view the code, find the reason.
  
  Although I have not configured the `url` of `backend_argument`, the code
  sets the default `url` values `CONF.cache.memcache_servers`(file
  `keystone/common/cache/core.py`, line 112-113).
  
  Therefore, the variable of `CONF.cache.memcache_servers` default value
  is `["localhost:11211"]`(file `keystone/common/config.py`, line
  356-359).
  
  So, when the code runs after `build_cache_conf` (file `keystone/common/cache/core.py`, line 139), the configuration of `cache` contain:
  ```
  url = ["localhost:11211"]
  host = "127.0.0.1"
  port = 6380
  db = 0
  ```
  
  I use the cache backend `dogpile.cache.redis`, when the code is
  configured the `url` argument and creates `redis_client`, will use the
  function `StrictRedis.from_url` and `url` argument, and ignore the
  argument `host`,`port` and `db`.
  
  If I will use the cache backend `dogpile.cache.redis`, I must configure the `keystone.conf`
  ```
  [cache]
  config_prefix = cache.keystone
  expiration_time = 600
  backend = dogpile.cache.redis
  backend_argument = url:redis://192.168.7.51:6380/0
  ```

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1618705

Title:
  keystone.cache.redis config arguments url conflict host and port

Status in OpenStack Identity (keystone):
  New

Bug description:
  version: 8.1.3.dev1
  branch: stable/liberty

  in my keystone.conf
  ```
  [cache]
  config_prefix = cache.keystone
  expiration_time = 600
  backend = dogpile.cache.redis
  backend_argument = host:127.0.0.1
  backend_argument = port:6380
  backend_argument = db:0
  ```

  Using this keystone.conf is not working.
  ```
  2016-08-31 13:46:56.552 10387 CRITICAL keystone [-] AttributeError: 'list' object has no attribute 'decode'
  2016-08-31 13:46:56.552 10387 ERROR keystone Traceback (most recent call last):
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/bin/keystone-wsgi-admin", line 34, in <module>
  2016-08-31 13:46:56.552 10387 ERROR keystone     server = wss.make_server('', args.port, initialize_admin_application())
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/wsgi.py", line 78, in initialize_admin_application
  2016-08-31 13:46:56.552 10387 ERROR keystone     return initialize_application('admin')
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/wsgi.py", line 64, in initialize_application
  2016-08-31 13:46:56.552 10387 ERROR keystone     startup_application_fn=loadapp)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/common.py", line 49, in setup_backends
  2016-08-31 13:46:56.552 10387 ERROR keystone     drivers = backends.load_backends()
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/server/backends.py", line 33, in load_backends
  2016-08-31 13:46:56.552 10387 ERROR keystone     cache.configure_cache_region(cache.REGION)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/keystone/common/cache/core.py", line 142, in configure_cache_region
  2016-08-31 13:46:56.552 10387 ERROR keystone     '%s.' % CONF.cache.config_prefix)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/region.py", line 552, in configure_from_config
  2016-08-31 13:46:56.552 10387 ERROR keystone     "%swrap" % prefix, None),
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/region.py", line 417, in configure
  2016-08-31 13:46:56.552 10387 ERROR keystone     _config_prefix
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/api.py", line 81, in from_config_dict
  2016-08-31 13:46:56.552 10387 ERROR keystone     for key in config_dict
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/backends/redis.py", line 109, in __init__
  2016-08-31 13:46:56.552 10387 ERROR keystone     self.client = self._create_client()
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/dogpile/cache/backends/redis.py", line 129, in _create_client
  2016-08-31 13:46:56.552 10387 ERROR keystone     return redis.StrictRedis.from_url(**args)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/redis/client.py", line 391, in from_url
  2016-08-31 13:46:56.552 10387 ERROR keystone     connection_pool = ConnectionPool.from_url(url, db=db, **kwargs)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/home/chuangwang/.ostack/lib/python3.5/site-packages/redis/connection.py", line 774, in from_url
  2016-08-31 13:46:56.552 10387 ERROR keystone     url = urlparse(url)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 294, in urlparse
  2016-08-31 13:46:56.552 10387 ERROR keystone     url, scheme, _coerce_result = _coerce_args(url, scheme)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 114, in _coerce_args
  2016-08-31 13:46:56.552 10387 ERROR keystone     return _decode_args(args) + (_encode_result,)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 98, in _decode_args
  2016-08-31 13:46:56.552 10387 ERROR keystone     return tuple(x.decode(encoding, errors) if x else '' for x in args)
  2016-08-31 13:46:56.552 10387 ERROR keystone   File "/usr/lib64/python3.5/urllib/parse.py", line 98, in <genexpr>
  2016-08-31 13:46:56.552 10387 ERROR keystone     return tuple(x.decode(encoding, errors) if x else '' for x in args)
  2016-08-31 13:46:56.552 10387 ERROR keystone AttributeError: 'list' object has no attribute 'decode'
  2016-08-31 13:46:56.552 10387 ERROR keystone
  ```

  Therefore, I view the code, find the reason.

  Although I have not configured the `url` of `backend_argument`, the
  code sets the default `url` values `CONF.cache.memcache_servers`(file
  `keystone/common/cache/core.py`, line 112-113).

  Therefore, the variable of `CONF.cache.memcache_servers` default value
  is `["localhost:11211"]`(file `keystone/common/config.py`, line
  356-359).

  So, when the code runs after `build_cache_conf` (file `keystone/common/cache/core.py`, line 139), the configuration of `cache` contain:
  ```
  url = ["localhost:11211"]
  host = "127.0.0.1"
  port = 6380
  db = 0
  ```

  I use the cache backend `dogpile.cache.redis`, when the code is
  configured the `url` argument and creates `redis_client`, will use the
  function `StrictRedis.from_url` and `url` argument, and ignore the
  argument `host`,`port` and `db`.

  If I will use the cache backend `dogpile.cache.redis`, I must configure the `keystone.conf`
  ```
  [cache]
  config_prefix = cache.keystone
  expiration_time = 600
  backend = dogpile.cache.redis
  backend_argument = url:redis://192.168.7.51:6380/0
  ```

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1618705/+subscriptions
Follow ups

[Bug 1618705] Re: keystone.cache.redis config arguments url conflict host and port
From: Lance Bragstad, 2017-07-25