openstack team mailing list archive

[Gabriel Hurley <Gabriel.Hurley@xxxxxxxxxx>]

Even though I don't experience this problem (and prefer nginx to apache), I can help diagnose:

Connections ending up in CLOSE_WAIT means that the socket isn't being fully closed, which is controlled by the client lib (in this case python-keystoneclient) which uses httplib2 under the hood.  When requests complete successfully httplib2 *does* close the connections just fine, so I'm wondering if you're actually triggering some kind of unhandled exception in keystoneclient. Are you seeing any errors in your logs anywhere? It's also worth noting that httplib2 has some very peculiar retry behaviors and other vagaries that come into play when the remote endpoint is unresponsive, etc.

Another potential problem is if you're running a proxy layer (such as haproxy) in the middle there are various configuration options which can cause the connection to remain open even after the "backend" has sent a complete response (adding inappropriate "keep-alive" headers, stripping "connection: close", filtering packets, etc.). The same is true of any other middleware you might be running that could get between the python process opening the socket and the remote end returning a response.

Hope something in there helps,

    - Gabriel

> -----Original Message-----
> From: openstack-bounces+gabriel.hurley=nebula.com@xxxxxxxxxxxxxxxxxxx
> [mailto:openstack-
> bounces+gabriel.hurley=nebula.com@xxxxxxxxxxxxxxxxxxx] On Behalf Of Sam
> Morrison
> Sent: Wednesday, January 30, 2013 7:36 PM
> To: openstack@xxxxxxxxxxxxxxxxxxx list
> Subject: [Openstack] Horizon and open connections
> 
> We have horizon running based on the Ubuntu Folsom Cloud Archive
> packages.
> 
> What I notice is that after a while we have thousands of connections in the
> CLOSE_WAIT state to keystone and our nova api servers.
> The host also uses up all it's available memory (2GB)
> 
> After a restart of apache all the connections are cleaned up and the memory
> used drops down to about 200MB
> 
> Just wondering if this is supposed to happen or is there a bug. It seems to me
> that horizon isn't closing connections or something.
> 
> Anyone have a similar issue/solution?
> 
> Cheers,
> Sam
> 
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp