openstack team mailing list archive

[Pádraig Brady <P@xxxxxxxxxxxxxx>]

On 07/29/2012 03:24 AM, Lars Kellogg-Stedman wrote:
> Our environment has connection-tracking firewalls that drop idle
> connections after an hour.  There is a connection between nova-compute
> and our qpidd server that appears to be idle for long periods of time.
> 
> When the firewall drops this connection, the participating hosts are
> unaware of that fact and ultimately stop communicating with each other
> until we restart nova-compute.
> 
> I was hoping that the qpid_heartbeat parameter would avoid this
> problem by keeping the connection active, but despite having
> qpid_heartbeat set explicitly in our configuration...
> 
>   # This is supposed to be the default
>   qpid_heartbeat = 5
> 
> ...there is no traffic across this connection
> 
> I can deal with this problem by forcing (via libkeepalive,
> http://libkeepalive.sourceforge.net) SO_KEEPALIVE on the AMQ sockets
> (and tuning the net.ipv4.tcp_keepalive_time sysctl to be < the
> firewall connection timeout), but that seems a bit of a hack.  It's
> also possible to work around this by disabling idle connection
> timeouts on the firewall, so we're not completely stymied...
> 
> ...but I would like to understand why setting qpid_heartbeat does not,
> in fact, result in the regular transmission of heartbeat packets
> across the connection.
> 
> We're running openstack-nova-2012.1.1-0.20120615.13614 from EPEL (and
> qpid 0.14).

Looks like a typo.
Could you try this.

cheers,
Pádraig.

diff --git a/nova/rpc/impl_qpid.py b/nova/rpc/impl_qpid.py
index 289f21b..e19079e 100644
--- a/nova/rpc/impl_qpid.py
+++ b/nova/rpc/impl_qpid.py
@@ -317,7 +317,7 @@ class Connection(object):
                     FLAGS.qpid_reconnect_interval_min)
         if FLAGS.qpid_reconnect_interval:
             self.connection.reconnect_interval = FLAGS.qpid_reconnect_interval
-        self.connection.hearbeat = FLAGS.qpid_heartbeat
+        self.connection.heartbeat = FLAGS.qpid_heartbeat
         self.connection.protocol = FLAGS.qpid_protocol
         self.connection.tcp_nodelay = FLAGS.qpid_tcp_nodelay