yahoo-eng-team team mailing list archive

[Dolph Mathews <1360446@xxxxxxxxxxxxxxxxxx>]

** Changed in: keystonemiddleware
       Status: Fix Committed => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1360446

Title:
  client connection leak to memcached under eventlet due to threadlocal

Status in OpenStack Identity (Keystone):
  Fix Committed
Status in Keystone icehouse series:
  Triaged
Status in OpenStack Identity  (Keystone) Middleware:
  Fix Released

Bug description:
  When Keystone configured with memcached as backend and token storage
  keystone didn't reuse connections to it and starting to fail after
  having more than 500 connections to the memcached.

  Steps to reproduce:

  1. Configure keystone with memcached as backend.
  2. Create moreless good load (creating of VM's creates a lot of connections) on keystone and watch for connections to memcached using netstat, ex. netstat -an |grep -c ":11211"

  Expected behavior:
  connections number should be reasonable and be not more that the number of connections to the keystone (ideally :)

  Observed bahavior:
  Number of connections growing and seems than
  1. They didn't reused at all.
  2. Lifetime of some connection is 600 seconds.
  3. It looks like not all the connections stay for 600 seconds.

  <UPDATE from MorganFainberg>
  This is specific to deploying under eventlet and the python-memcached library and it's explicit/unavoidable use of threadlocal. Use of threadlocal under eventlet causes the client connections to leak until the GC / kernel cleans up the connections. This was confirmed to only affect eventlet with patch threading.

  Keystone deployed under apache is not affected.

  All services deployed with keystonemiddleware that utilize eventlet
  and memcache for token cache are also affected.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1360446/+subscriptions