yahoo-eng-team team mailing list archive

[Morgan Fainberg <morgan.fainberg@xxxxxxxxx>]

Based upon the conversations in IRC and looking into this it appears
that what is going on here is a thread-local based cache (in memory) of
the tokens.

This is not something that really can be fixed within keystonemiddleware
without providing a cross-thread/non-thread.local dictionary cache (in
process). This is specifically where using an external service such as
memcached will ensure coherency in the cache.

This issue results from the following scenario (more detail):

2 Processes, each running keystonemiddleware with local caching of the tokens.
1 of these caches the token, the other does not.
the cached version succeeds, the token is valid
the non-cached version calls back to keystone, and due to clock-skew the keystone server reports the token as invalid.

This now places the system into a state where one endpoint will result
in success and the other will result in invalid token.

The solution is to use memcached or another external system that
both/all processes can reference for caching the tokens. Utilizing NTP
also mitigates the issue as all clocks appear close enough now that the
tokens would be reported in all endpoints as no-longer valid or valid
(depending on the direction of the clock-skew).

** Changed in: keystone
       Status: New => Invalid

** Changed in: keystonemiddleware
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1415687

Title:
  Keystone middleware intermittently rejects a token based on expiration

Status in OpenStack Identity (Keystone):
  Invalid
Status in OpenStack Identity  (Keystone) Middleware:
  Invalid

Bug description:
  I noticed an issue that is effected by a system clock skew that
  results in keystone, via a service using keystone-middleware, to
  intermittently succeed/fail/succeed/fail/etc, requests that use the
  same token. This issue is related to the system clock in some way as
  installing ntp and forcing a clock update immediately resolved the
  issue.

  Output from the client's perspective (curl requests against Trove,
  using keystone-middleware):

  http://paste.openstack.org/show/163419/

  Output from Trove's log, including middleware logging:

  http://paste.openstack.org/show/163427/

  Output from keystone.log:

  http://paste.openstack.org/show/163421/

  Steps to reproduce:

  New VM (Ubuntu in my case)
  Ensure that the system clock lags a few hours
  Install devstack, enabling trove
  Get a token from keystone for the admin tenant
  curl trove /instances for the admin tenant using the token
  Repeat the curl numerous times. Requests should start to succeed/fail intermittently.

  To resolve, install and start ntp.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1415687/+subscriptions