sws team mailing list archive

[Launchpad Bug Tracker <1020127@xxxxxxxxxxxxxxxxxx>]

Alex Yang (alexyang) has assigned this bug to you for Keystone:

Hello,

The configuration for caching auth tokens in Swift described at 
    https://lists.launchpad.net/openstack/msg08639.html
where
    memcache_servers = cachedev1:11211
is added to the [filter:authtoken] section of /etc/swift/proxy-server.conf was applied to 2 Swift proxy servers.  After this a Swift reliability test consisting of 50 threads performing repeated PUT and GET requests started to intermittently receive 500 Internal Server Error responses and the Swift log on the proxy servers reported the following messages, each one corresponding to each 500 response.  The messages continue to occur intermittently, sometimes for read and sometimes for write.  

Taking this configuration out again and restarting the proxy servers
prevents the problem.

The Swift version is 1.4.8 (Essex).

The Swift reliability test is as described in 
    https://answers.launchpad.net/swift/+question/201627
but currently rate limited to 1,050 HTTP requests/min.  

Jul  2 10:19:38 swiftproxydev2 <131>proxy-server Error: Second
simultaneous write on fileno 15 detected.  Unless you really know what
you're doing, make sure that only one greenthread can write any
particular socket.  Consider using a pools.Pool. If you do know what
you're doing and want to disable this error, call
eventlet.debug.hub_multiple_reader_prevention(False): #012Traceback
(most recent call last):#012  File "/usr/lib/python2.6/site-
packages/swift/common/middleware/catch_errors.py", line 47, in
__call__#012    return self.app(env, my_start_response)#012  File
"/usr/lib/python2.6/site-
packages/swift/common/middleware/healthcheck.py", line 38, in
__call__#012    return self.app(env, start_response)#012  File
"/usr/lib/python2.6/site-packages/swift/common/middleware/memcache.py",
line 47, in __call__#012    return self.app(env, start_response)#012
File "/usr/lib/python2.6/site-
packages/keystone/middleware/auth_token.py", line 173, in __call__#012
token_info = self._validate_user_token(user_token)#012  File
"/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py",
line 343, in _validate_user_token#012    cached =
self._cache_get(user_token)#012  File "/usr/lib/python2.6/site-
packages/keystone/middleware/auth_token.py", line 477, in _cache_get#012
cached = self._cache.get(key)#012  File "/usr/lib/python2.6/site-
packages/memcache.py", line 661, in get#012    server.send_cmd("get %s"
% key)#012  File "/usr/lib/python2.6/site-packages/memcache.py", line
873, in send_cmd#012    self.socket.sendall(cmd + '\r\n')#012  File
"/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 283, in
sendall#012    tail = self.send(data, flags)#012  File
"/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 278, in
send#012    timeout_exc=socket.timeout("timed out"))#012  File
"/usr/lib/python2.6/site-packages/eventlet/hubs/__init__.py", line 119,
in trampoline#012    listener = hub.add(hub.WRITE, fileno,
current.switch)#012  File "/usr/lib/python2.6/site-
packages/eventlet/hubs/poll.py", line 27, in add#012    listener =
super(

Jul  2 10:24:15 swiftproxydev2 <131>proxy-server Error: Second
simultaneous read on fileno 15 detected.  Unless you really know what
you're doing, make sure that only one greenthread can read any
particular socket.  Consider using a pools.Pool. If you do know what
you're doing and want to disable this error, call
eventlet.debug.hub_multiple_reader_prevention(False): #012Traceback
(most recent call last):#012  File "/usr/lib/python2.6/site-
packages/swift/common/middleware/catch_errors.py", line 47, in
__call__#012    return self.app(env, my_start_response)#012  File
"/usr/lib/python2.6/site-
packages/swift/common/middleware/healthcheck.py", line 38, in
__call__#012    return self.app(env, start_response)#012  File
"/usr/lib/python2.6/site-packages/swift/common/middleware/memcache.py",
line 47, in __call__#012    return self.app(env, start_response)#012
File "/usr/lib/python2.6/site-
packages/keystone/middleware/auth_token.py", line 173, in __call__#012
token_info = self._validate_user_token(user_token)#012  File
"/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py",
line 343, in _validate_user_token#012    cached =
self._cache_get(user_token)#012  File "/usr/lib/python2.6/site-
packages/keystone/middleware/auth_token.py", line 477, in _cache_get#012
cached = self._cache.get(key)#012  File "/usr/lib/python2.6/site-
packages/memcache.py", line 662, in get#012    rkey, flags, rlen, =
self._expectvalue(server)#012  File "/usr/lib/python2.6/site-
packages/memcache.py", line 747, in _expectvalue#012    line =
server.readline()#012  File "/usr/lib/python2.6/site-
packages/memcache.py", line 886, in readline#012    data =
recv(4096)#012  File "/usr/lib/python2.6/site-
packages/eventlet/greenio.py", line 238, in recv#012
timeout_exc=socket.timeout("timed out"))#012  File "/usr/lib/python2.6
/site-packages/eventlet/hubs/__init__.py", line 117, in trampoline#012
listener = hub.add(hub.READ, fileno, current.switch)#012  File
"/usr/lib/python2.6/site-packages/eventlet/hubs/poll.py", line 27, in
add#012    listener = super(Hub, self).a

A problem producing a similar error in Keystone is discussed at
    https://bugs.launchpad.net/keystone/+bug/1012381
but I don't know if this issue is related.

** Affects: keystone
     Importance: High
     Assignee: Sina Web Service Dev (sws)
         Status: Triaged

-- 
proxy-server Error: Second simultaneous read or write detected
https://bugs.launchpad.net/bugs/1020127
You received this bug notification because you are a member of Sina Web Service Dev, which is a bug assignee.