yahoo-eng-team team mailing list archive

[Lance Bragstad <lbragstad@xxxxxxxxx>]

Matt, are you able to recreate this with master or mitaka?

** Also affects: keystone/liberty
   Importance: Undecided
       Status: New

** Summary changed:

- memcache pool reap issue (stable/liberty)
+ memcache pool reap issue

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1570158

Title:
  memcache pool reap issue

Status in OpenStack Identity (keystone):
  New
Status in OpenStack Identity (keystone) liberty series:
  New

Bug description:
  There seems to be a code error in the memcache pool cleanup. I'm
  seeing this on stable/liberty (built as of 2 weeks ago). I don't have
  a specific reproducer for this, it just seems to happen. Looking at
  the code I don't really understand how this can happen.

  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi [req-30e23a7d-754d-4a89-904f-1804a20029f2 - - - - -] deque index out of range
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi Traceback (most recent call last):
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/keystone/common/wsgi.py", line 248, in __call__
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     result = method(context, **params)
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/keystone/common/controller.py", line 138, in inner
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     context['subject_token_id']))
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/keystone/token/provider.py", line 188, in validate_token
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     token = self._validate_token(unique_id)
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/dogpile/cache/region.py", line 1053, in decorate
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     should_cache_fn)
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/dogpile/cache/region.py", line 657, in get_or_create
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     async_creator) as value:
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 158, in __enter__
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     return self._enter()
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/dogpile/core/dogpile.py", line 91, in _enter
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     value = value_fn()
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/dogpile/cache/region.py", line 610, in get_value
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     value = self.backend.get(key)
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/dogpile/cache/backends/memcached.py", line 161, in get
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     value = self.client.get(key)
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/keystone/common/cache/backends/memcache_pool.py", line 36, in _run_method
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     return getattr(client, __name)(*args, **kwargs)
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     self.gen.next()
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/keystone/common/cache/_memcache_pool.py", line 132, in acquire
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     self._drop_expired_connections()
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi   File "/venv/local/lib/python2.7/site-packages/keystone/common/cache/_memcache_pool.py", line 166, in _drop_expired_connections
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi     while self.queue and self.queue[0].ttl < now:
  2016-04-13 20:50:29.124 46 ERROR keystone.common.wsgi IndexError: deque index out of range

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1570158/+subscriptions