yahoo-eng-team team mailing list archive

[Morgan Fainberg <morgan.fainberg@xxxxxxxxx>]

This affects any server using keystonemiddleware that runs under
eventlet and uses memcache for token caching. The fix for
keystonemiddleware will likely be based on the Keystone fix.

** Also affects: keystonemiddleware
   Importance: Undecided
       Status: New

** Changed in: keystonemiddleware
   Importance: Undecided => High

** Changed in: keystonemiddleware
       Status: New => Triaged

** Tags added: icehouse-backport-potential

** Also affects: keystone/icehouse
   Importance: Undecided
       Status: New

** Changed in: keystone/icehouse
       Status: New => Triaged

** Changed in: keystone/icehouse
   Importance: Undecided => Medium

** Summary changed:

- Keystone  server didn't reuse connections to memcached
+ client connection leak to memcached under eventlet due to threadlocal

** Description changed:

  When Keystone configured with memcached as backend and token storage
  keystone didn't reuse connections to it and starting to fail after
  having more than 500 connections to the memcached.
  
  Steps to reproduce:
  
  1. Configure keystone with memcached as backend.
  2. Create moreless good load (creating of VM's creates a lot of connections) on keystone and watch for connections to memcached using netstat, ex. netstat -an |grep -c ":11211"
  
  Expected behavior:
  connections number should be reasonable and be not more that the number of connections to the keystone (ideally :)
  
  Observed bahavior:
  Number of connections growing and seems than
  1. They didn't reused at all.
  2. Lifetime of some connection is 600 seconds.
  3. It looks like not all the connections stay for 600 seconds.
+ 
+ <UPDATE from MorganFainberg>
+ This is specific to deploying under eventlet and the python-memcached library and it's explicit/unavoidable use of threadlocal. Use of threadlocal under eventlet causes the client connections to leak until the GC / kernel cleans up the connections. This was confirmed to only affect eventlet with patch threading.
+ 
+ Keystone deployed under apache is not affected.
+ 
+ All services deployed with keystonemiddleware that utilize eventlet and
+ memcache for token cache are also affected.

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1360446

Title:
  client connection leak to memcached under eventlet due to threadlocal

Status in OpenStack Identity (Keystone):
  Triaged
Status in Keystone icehouse series:
  Triaged
Status in OpenStack Identity  (Keystone) Middleware:
  Triaged

Bug description:
  When Keystone configured with memcached as backend and token storage
  keystone didn't reuse connections to it and starting to fail after
  having more than 500 connections to the memcached.

  Steps to reproduce:

  1. Configure keystone with memcached as backend.
  2. Create moreless good load (creating of VM's creates a lot of connections) on keystone and watch for connections to memcached using netstat, ex. netstat -an |grep -c ":11211"

  Expected behavior:
  connections number should be reasonable and be not more that the number of connections to the keystone (ideally :)

  Observed bahavior:
  Number of connections growing and seems than
  1. They didn't reused at all.
  2. Lifetime of some connection is 600 seconds.
  3. It looks like not all the connections stay for 600 seconds.

  <UPDATE from MorganFainberg>
  This is specific to deploying under eventlet and the python-memcached library and it's explicit/unavoidable use of threadlocal. Use of threadlocal under eventlet causes the client connections to leak until the GC / kernel cleans up the connections. This was confirmed to only affect eventlet with patch threading.

  Keystone deployed under apache is not affected.

  All services deployed with keystonemiddleware that utilize eventlet
  and memcache for token cache are also affected.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1360446/+subscriptions