yahoo-eng-team team mailing list archive

[Alexander Makarov <amakarov@xxxxxxxxxxxx>]

Public bug reported:

At some point Keystone stops processing HTTP requests and HAProxy moves
the server out of pool. Once Keystones on all controllers stuck HAProxy
starts to respond with HTTP 503 error.

Symptoms:
~~~~~~~~~
HAProxy log:
2014-12-08T11:01:06.135849+00:00 info: 192.168.0.23:34984 [08/Dec/2014:11:01:06.135] keystone-2 keystone-2/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 366/0/0/0/0 0/0 "POST /v2.0/tokens HTTP/1.1"

Keystone keeps accepting TCP connection, but hangs at HTTP level
(meaning that telnet works, but curl hangs)

root@node-21:/var/log/keystone# strace -p 26330
Process 26330 attached - interrupt to quit
futex(0xe50420, FUTEX_WAIT_PRIVATE, 0, NULL

The last logged message from stuck process:
2014-12-05 17:01:20.749 26330 DEBUG keystone.common.cache._memcache_pool [-] Memcached pool 39391504, thread 139781728069376: Acquiring connection _debug_logger /usr/lib/python2.7/dist-packages/keystone/common/cache/_memcache_pool.py:91

Memcache pool has asymmetric behaviour on aquire and release of a connection:
If pool is depleted a new connection is created
If pool has reached it's maximum size that connection excess still being pushed back to the pool causing thread lock.

** Affects: keystone
     Importance: Undecided
     Assignee: Alexander Makarov (amakarov)
         Status: New

** Changed in: keystone
     Assignee: (unassigned) => Alexander Makarov (amakarov)

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1401108

Title:
  Memcache pool locks on release

Status in OpenStack Identity (Keystone):
  New

Bug description:
  At some point Keystone stops processing HTTP requests and HAProxy
  moves the server out of pool. Once Keystones on all controllers stuck
  HAProxy starts to respond with HTTP 503 error.

  Symptoms:
  ~~~~~~~~~
  HAProxy log:
  2014-12-08T11:01:06.135849+00:00 info: 192.168.0.23:34984 [08/Dec/2014:11:01:06.135] keystone-2 keystone-2/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 366/0/0/0/0 0/0 "POST /v2.0/tokens HTTP/1.1"

  Keystone keeps accepting TCP connection, but hangs at HTTP level
  (meaning that telnet works, but curl hangs)

  root@node-21:/var/log/keystone# strace -p 26330
  Process 26330 attached - interrupt to quit
  futex(0xe50420, FUTEX_WAIT_PRIVATE, 0, NULL

  The last logged message from stuck process:
  2014-12-05 17:01:20.749 26330 DEBUG keystone.common.cache._memcache_pool [-] Memcached pool 39391504, thread 139781728069376: Acquiring connection _debug_logger /usr/lib/python2.7/dist-packages/keystone/common/cache/_memcache_pool.py:91

  Memcache pool has asymmetric behaviour on aquire and release of a connection:
  If pool is depleted a new connection is created
  If pool has reached it's maximum size that connection excess still being pushed back to the pool causing thread lock.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1401108/+subscriptions