openstack team mailing list archive

[Roman Sokolkov <rsokolkov@xxxxxxxxx>]

---------- Forwarded message ----------
From: Roman Sokolkov <rsokolkov@xxxxxxxxx>
Date: 2011/12/28
Subject: Re: [Openstack] reconnecting to RabbitMQ
To: Ask Solem <ask@xxxxxxxxxxxx>


Hi!

I tried to make Rabbit HA for Openstack.

My first e-mail was little innacurate.

I use two nodes with active/active rabbits with mirrored queues.
Additionally I use Virtual IP. And all it managed by Pacemaker. Openstack
uses this remote rabbitmq via VIP.

I hardly reboot node with VIP and VIP moves to other node. In this case,
connections on Openstack host staying forever in ESTABLISHED state(in
py-amqlib 0.6.1 no tcp keepalive options).
Openstack think that all good and doesn't try to reconnect. But I if
correctly stop RabbitMQ, connections closed correctly and Openstack will
try to reconnect.

I solve this issue by adding tcp_keepalive options in
py-amqlib(transport.py).

It's very interesting what implemented in KAMQP. And how heartbeats work?



2011/12/28 Ask Solem <ask@xxxxxxxxxxxx>

> Not sure how I can help until the problem is identified.
> Kombu does not do automatic reconnecting by default, but provides
> the primitives necessary to implement it.
>
> amqplib does not support heartbeats so there may be cases
> where the client/server does not properly detect that the socket
> has been closed, this usually only a problem with some particular
> intermediate
> network components like routers and firewalls.  Support for heartbeats
> is currently in the works, and I already have a rudimentary implementation
> at http://github.com/ask/kamqp/ if someone wants to try it out (though
> that
> probably requires some hacking).
>
>
> On 28 Dec 2011, at 10:51, Alexis Richardson wrote:
>
> > Hi all,
> >
> > I am bcc'ing the author of Kombu in case he can help.
> >
> > alexis
> >
> >
> > On Sun, Dec 25, 2011 at 2:11 AM, Chris Behrens
> > <chris.behrens@xxxxxxxxxxxxx> wrote:
> >> Hi Roman,
> >>
> >> Are you using diablo or trunk (both of which use kombu by default and
> should support reconnecting)?
> >>
> >> If so, can you 'egrep' for the following strings in logs and provide
> the output?  You can reply to me directly so we're not spamming this list.
> >>
> >> "Unable to connect to AMQP server"
> >> "Connected to AMQP server"
> >> "Re-established AMQP queues"
> >> "AMQP server on.*unreachable"
> >> 'Failed to consume"
> >> "Failed to publish"
> >>
> >> Thanks,
> >>
> >> - Chris
> >>
> >>
> >> On Dec 24, 2011, at 1:56 PM, Roman Sokolkov wrote:
> >>
> >>> Hello!
> >>>
> >>> If for some reason rabbitmq has been down some time or connection was
> aborted, nova couldn't reconnect automatically and I need to restart all
> nova services manually.
> >>>
> >>> Additionally after restarting rabbit, If I make some call to API. For
> example, "nova floating-ip-list" only
> >>> API reconnects automatically. Other nova services staying disconnected.
> >>>
> >>> Are there mechanisms to monitor establising of connections? And it
> difficult to add them?
> >>>
> >>> --
> >>> Best regards, Roman Sokolkov
> >>> _______________________________________________
> >>> Mailing list: https://launchpad.net/~openstack
> >>> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >>> Unsubscribe : https://launchpad.net/~openstack
> >>> More help   : https://help.launchpad.net/ListHelp
> >>
> >>
> >> _______________________________________________
> >> Mailing list: https://launchpad.net/~openstack
> >> Post to     : openstack@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~openstack
> >> More help   : https://help.launchpad.net/ListHelp
>
>


-- 
Regards, Roman Sokolkov



-- 
Regards, Roman Sokolkov