fuel-dev team mailing list archive

[Vladimir Kuklin <vkuklin@xxxxxxxxxxxx>]

It will not help if you shut down the controller. The problem is that you
have  hanged AMQP sessions which kombu driver does not look to handle
correctly.


On Fri, Feb 28, 2014 at 1:09 PM, Bogdan Dobrelya <bdobrelia@xxxxxxxxxxxx>wrote:

> On 02/28/2014 05:44 AM, Dmitry Borodaenko wrote:
> > Team,
> >
> > Me and Ryan have spent all day investigating
> > https://bugs.launchpad.net/fuel/+bug/1285449
> >
> > What we have found so far confirms that this is a critical bug that
> > absolutely must be resolved before 4.1 is released.  I have documented
> > our findings in the bug comments, someone please take over the
> > investigation when you come to the office tomorrow morning MSK time.
> >
> > I have a feeling that once the root cause is found, the fix will be
> > low-impact and will involve either change in HAProxy configuration for
> > RabbitMQ, a patch/upgrade of HAProxy or kombu, or something similar.
> > But first we need to understand what exactly breaks, and why this only
> > affects some services and not all of them.
> >
> > Thanks,
> >
>
> Here is recent rabbitMQ discussion quote from the
> Fuel-conductors-support team skype chat (RU + translation):
>
> Wednesday, February 26, 2014
> [4:00:10 PM] Maxim Yefimov: Коллеги, вопрос есть:
> (I have a question)
>
> listen rabbitmq-openstack
>   bind 192.168.0.2:5672
>   balance  roundrobin
>
>   server  node-1 192.168.0.3:5673   check inter 5000 rise 2 fall 3
>   server  node-2 192.168.0.4:5673   check inter 5000 rise 2 fall 3  backup
>   server  node-3 192.168.0.5:5673   check inter 5000 rise 2 fall 3  backup
>
> [4:01:01 PM] Maxim Yefimov: Зачем одновременно roundrobin и active-passive?
> (Why do we use roundrobin and active-passive at once for RabbitMQ?)
>
> [4:01:39 PM] Miroslav Anashkin: Чтобы коннект не рвался
> (To make sure the connection wouldn't break)
>
> [4:02:01 PM] Miroslav Anashkin: У кролика кластер существует строго в
> виде мастер-слейв
> (RabbitMQ clustering is restricted to master-slave only)
>
> [4:02:23 PM] Miroslav Anashkin: Соответственно даже если какая-то нода с
> запросом к слейву придет - та его на мастер отправит
> (Hence, any node's query to the RabbitMQ slave would have been re-sent
> to the master)
>
> [4:02:52 PM] Miroslav Anashkin: Поэтому сделали так чтобы ХАПрокси
> всегда всех посылал на одну ноду
> (Thats why HAproxy always redirects all queries to the single RabbitMQ
> node)
>
> And I'm not clear with this explanation, honestly. Why couldn't we make
> OS establish direct connections to arbitrary (LB) chosen RabbitMQ nodes
> skipping HAproxy at all? (because of this: "any node's query to the
> RabbitMQ slave would have been re-sent to the master")
>
> Could that resolve the issue? I think I will investigate this option as
> well.
>
>
> --
> Best regards,
> Bogdan Dobrelya,
> Skype #bogdando_at_yahoo.com
> Irc #bogdando
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>



-- 
Yours Faithfully,
Vladimir Kuklin,
Senior Deployment Engineer,
Mirantis, Inc.
+7 (495) 640-49-04
+7 (926) 702-39-68
Skype kuklinvv
45bk3, Vorontsovskaya Str.
Moscow, Russia,
www.mirantis.com <http://www.mirantis.ru/>
www.mirantis.ru
vkuklin@xxxxxxxxxxxx