yahoo-eng-team team mailing list archive

[Shawn Berger <slberger@xxxxxxxxxx>]

Public bug reported:

With memcache set up for resource caching, when a tenant is created,
deleted, and recreated with the same name, users within that project get
intermittent errors when requesting tokens.

You can recreate this by having memcache with resource caching enabled.
Then create a tenant, delete it, and then recreate it making sure the
name is the same as the first one.  Then create a user in this tenant
and continually request tokens.  It will gradually start generating
tokens while also failing until the cache is cleaned out.

I believe the intermittent errors we experienced were due to our
environment having a memcache on each keystone node and having the
keystone nodes behind a load balancer.

As I ran this scenario, I was seeing more failures in the beginning and
then it gradually started having more successes until a little after the
cache expiration_time where I was seeing all successes.

We investigated and when this error was originally hit it threw 404 or
401s.  The 404s were complaining about not being able to find a certain
project, but when I tried to recreate I was receiving all 401s.

The 404 errors led me to believe that this was due to memcache not
marking cache entries as deleted.  Since, when running our tests we used
the name of the project and it would auto resolve the id.  So the entry
for the project name in the cache was conflicting with the entry in the
database, but once the cache is expired it isn't an issue.

So it seems that reusing names of projects causes problems with the
resolution of the project id when memcache is enabled.

** Affects: keystone
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1504686

Title:
  Keystone errors on token requests for users in recreated tenants when
  using memcache

Status in Keystone:
  New

Bug description:
  With memcache set up for resource caching, when a tenant is created,
  deleted, and recreated with the same name, users within that project
  get intermittent errors when requesting tokens.

  You can recreate this by having memcache with resource caching
  enabled.  Then create a tenant, delete it, and then recreate it making
  sure the name is the same as the first one.  Then create a user in
  this tenant and continually request tokens.  It will gradually start
  generating tokens while also failing until the cache is cleaned out.

  I believe the intermittent errors we experienced were due to our
  environment having a memcache on each keystone node and having the
  keystone nodes behind a load balancer.

  As I ran this scenario, I was seeing more failures in the beginning
  and then it gradually started having more successes until a little
  after the cache expiration_time where I was seeing all successes.

  We investigated and when this error was originally hit it threw 404 or
  401s.  The 404s were complaining about not being able to find a
  certain project, but when I tried to recreate I was receiving all
  401s.

  The 404 errors led me to believe that this was due to memcache not
  marking cache entries as deleted.  Since, when running our tests we
  used the name of the project and it would auto resolve the id.  So the
  entry for the project name in the cache was conflicting with the entry
  in the database, but once the cache is expired it isn't an issue.

  So it seems that reusing names of projects causes problems with the
  resolution of the project id when memcache is enabled.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1504686/+subscriptions