yahoo-eng-team team mailing list archive

[Adam Young <1020127@xxxxxxxxxxxxxxxxxx>]

No fix necessary in Keystone server

** Changed in: keystone
       Status: In Progress => Fix Released

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Keystone.
https://bugs.launchpad.net/bugs/1020127

Title:
  proxy-server Error: Second simultaneous read or write detected

Status in OpenStack Identity (Keystone):
  Fix Released
Status in Python client library for Keystone:
  Fix Released

Bug description:
  Hello,

  The configuration for caching auth tokens in Swift described at 
      https://lists.launchpad.net/openstack/msg08639.html
  where
      memcache_servers = cachedev1:11211
  is added to the [filter:authtoken] section of /etc/swift/proxy-server.conf was applied to 2 Swift proxy servers.  After this a Swift reliability test consisting of 50 threads performing repeated PUT and GET requests started to intermittently receive 500 Internal Server Error responses and the Swift log on the proxy servers reported the following messages, each one corresponding to each 500 response.  The messages continue to occur intermittently, sometimes for read and sometimes for write.  

  Taking this configuration out again and restarting the proxy servers
  prevents the problem.

  The Swift version is 1.4.8 (Essex).

  The Swift reliability test is as described in 
      https://answers.launchpad.net/swift/+question/201627
  but currently rate limited to 1,050 HTTP requests/min.  

  Jul  2 10:19:38 swiftproxydev2 <131>proxy-server Error: Second
  simultaneous write on fileno 15 detected.  Unless you really know what
  you're doing, make sure that only one greenthread can write any
  particular socket.  Consider using a pools.Pool. If you do know what
  you're doing and want to disable this error, call
  eventlet.debug.hub_multiple_reader_prevention(False): #012Traceback
  (most recent call last):#012  File "/usr/lib/python2.6/site-
  packages/swift/common/middleware/catch_errors.py", line 47, in
  __call__#012    return self.app(env, my_start_response)#012  File
  "/usr/lib/python2.6/site-
  packages/swift/common/middleware/healthcheck.py", line 38, in
  __call__#012    return self.app(env, start_response)#012  File
  "/usr/lib/python2.6/site-
  packages/swift/common/middleware/memcache.py", line 47, in
  __call__#012    return self.app(env, start_response)#012  File
  "/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py",
  line 173, in __call__#012    token_info =
  self._validate_user_token(user_token)#012  File "/usr/lib/python2.6
  /site-packages/keystone/middleware/auth_token.py", line 343, in
  _validate_user_token#012    cached = self._cache_get(user_token)#012
  File "/usr/lib/python2.6/site-
  packages/keystone/middleware/auth_token.py", line 477, in
  _cache_get#012    cached = self._cache.get(key)#012  File
  "/usr/lib/python2.6/site-packages/memcache.py", line 661, in get#012
  server.send_cmd("get %s" % key)#012  File "/usr/lib/python2.6/site-
  packages/memcache.py", line 873, in send_cmd#012
  self.socket.sendall(cmd + '\r\n')#012  File "/usr/lib/python2.6/site-
  packages/eventlet/greenio.py", line 283, in sendall#012    tail =
  self.send(data, flags)#012  File "/usr/lib/python2.6/site-
  packages/eventlet/greenio.py", line 278, in send#012
  timeout_exc=socket.timeout("timed out"))#012  File "/usr/lib/python2.6
  /site-packages/eventlet/hubs/__init__.py", line 119, in trampoline#012
  listener = hub.add(hub.WRITE, fileno, current.switch)#012  File
  "/usr/lib/python2.6/site-packages/eventlet/hubs/poll.py", line 27, in
  add#012    listener = super(

  Jul  2 10:24:15 swiftproxydev2 <131>proxy-server Error: Second
  simultaneous read on fileno 15 detected.  Unless you really know what
  you're doing, make sure that only one greenthread can read any
  particular socket.  Consider using a pools.Pool. If you do know what
  you're doing and want to disable this error, call
  eventlet.debug.hub_multiple_reader_prevention(False): #012Traceback
  (most recent call last):#012  File "/usr/lib/python2.6/site-
  packages/swift/common/middleware/catch_errors.py", line 47, in
  __call__#012    return self.app(env, my_start_response)#012  File
  "/usr/lib/python2.6/site-
  packages/swift/common/middleware/healthcheck.py", line 38, in
  __call__#012    return self.app(env, start_response)#012  File
  "/usr/lib/python2.6/site-
  packages/swift/common/middleware/memcache.py", line 47, in
  __call__#012    return self.app(env, start_response)#012  File
  "/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py",
  line 173, in __call__#012    token_info =
  self._validate_user_token(user_token)#012  File "/usr/lib/python2.6
  /site-packages/keystone/middleware/auth_token.py", line 343, in
  _validate_user_token#012    cached = self._cache_get(user_token)#012
  File "/usr/lib/python2.6/site-
  packages/keystone/middleware/auth_token.py", line 477, in
  _cache_get#012    cached = self._cache.get(key)#012  File
  "/usr/lib/python2.6/site-packages/memcache.py", line 662, in get#012
  rkey, flags, rlen, = self._expectvalue(server)#012  File
  "/usr/lib/python2.6/site-packages/memcache.py", line 747, in
  _expectvalue#012    line = server.readline()#012  File
  "/usr/lib/python2.6/site-packages/memcache.py", line 886, in
  readline#012    data = recv(4096)#012  File "/usr/lib/python2.6/site-
  packages/eventlet/greenio.py", line 238, in recv#012
  timeout_exc=socket.timeout("timed out"))#012  File "/usr/lib/python2.6
  /site-packages/eventlet/hubs/__init__.py", line 117, in trampoline#012
  listener = hub.add(hub.READ, fileno, current.switch)#012  File
  "/usr/lib/python2.6/site-packages/eventlet/hubs/poll.py", line 27, in
  add#012    listener = super(Hub, self).a

  A problem producing a similar error in Keystone is discussed at
      https://bugs.launchpad.net/keystone/+bug/1012381
  but I don't know if this issue is related.

To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1020127/+subscriptions