| Thread Previous • Date Previous • Date Next • Thread Next |
On 01/31/2013 11:59 AM, Gabriel Hurley wrote:
Even though I don't experience this problem (and prefer nginx to apache), I can help diagnose: Connections ending up in CLOSE_WAIT means that the socket isn't being fully closed, which is controlled by the client lib (in this case python-keystoneclient) which uses httplib2 under the hood.
Expanding on that a bit. CLOSE_WAIT is the state a TCP endpoint will enter upon receiving a FINished segment from the remote TCP. When the FIN arrives, the local application will receive notification of this via the classic "read return of zero" on a receive/read call against the socket. The FIN segment "means" "I will be sending you no more data."
Meanwhile, the local TCP will have ACKed the FIN segment, and the remote TCP will transition to FIN_WAIT_2 upon receipt of that ACK (until then it will be in FIN_WAIT_1).
Depending on how the remote application triggered the sending of the FIN, the TCP connection is now in a perfectly valid simplex state wherein the side in CLOSE_WAIT can continue sending data to the side which will now be in FIN_WAIT_2. It is exceedingly rare for applications to want a simplex TCP connection
If such a unidirectional TCP connection is not of any use to an application, (the common case)then that application should/must also close the connection upon the read return of zero.
Thus, seeing lots of connections "stuck" in CLOSE_WAIT is an indication of an application-level (relative to TCP) bug wherein the application on the "CLOSE_WAIT side" is ignoring the read return of zero.
Such bugs in applications may be "masked" by a few things:1) If the remote side called close() rather than shutdown(SHUT_WR) then an attempt on the CLOSE_WAIT side to send data to the remote will cause the remote TCP to return a RST segment (reset) because there is no longer anything above TCP to receive the data. This will then cause the local TCP to terminate the connection. This may also happen if the local application set SO_KEEPALIVE to enable TCP keepalives.
*) If the local side doesn't send anything, and doesn't have TCP keepalives set, if the remote TCP has a FIN_WAIT_2 timer of some sort going (long story involving a hole in the TCP specification and implementation workarounds, email if you want to hear it) then when that FIN_WAIT_2 timer expires the remote TCP may sent a RST segment.
RST segments are "best effort" in sending - they don't get retransmitted explicitly. In case 1 if the RST segment doesn't make it back, the local TCP will retransmit the data it was sending (because it will not have received an ACKnowledgement either). It will then either receive the RST triggered by that retransmission, or if no RSTs ever make it back, the local TCP will at some point reach its retransmission limit and terminate the connection. In case 2, if that one RST is lost, that's it, and the CLOSE_WAIT may remain forever.
Again though, given the rarity of actual application use of a simplex TCP connection, 99 times out of 10, seeing lots of CLOSE_WAIT connections building-up implies a buggy application or the libraries doing work on its behalf.
rick jones
| Thread Previous • Date Previous • Date Next • Thread Next |
