fuel-dev team mailing list archive
-
fuel-dev team
-
Mailing list archive
-
Message #00694
Re: Issues with RabbitMQ update
Great findings, Lukasz!
Adding larger audience of fuel-dev..
On Tue, Mar 25, 2014 at 3:37 PM, Lukasz Oles <loles@xxxxxxxxxxxx> wrote:
> Vladimir,
>
> there is no ISO, just install newest rabbitmq. I attached rpm package for
> centos.
>
> I have done some more investigation and number of workers actually doesn't
> matter. It just gives random results, but I think I found solution.
>
> Naily is using asynchronous library to communicate with Rabbitmq. It uses
> amqp library which uses EventMachine. In Naily event loop is running in
> main thread but consumer is running in another and publisher in yet another
> thread.
>
> To solve the problem with hanging I moved the code for publisher and
> consumer to EM::next_tick block. After this everything is working again
> now.
> EM::next_tick does two things. First, it schedules code to run in next
> event loop iteration. Second it runs this code in event loop thread. I'm
> not sure which of this things helps.
> Debugging async code in threads is really hard. Why in the first place
> Naily is using async library?
>
> What do you think about it? Maybe it would be better just move to
> synchronous library like bunny?
>
> Please remember I'm not ruby programmer so I can be missing something here.
>
> Regards
>
>
> On Tue, Mar 25, 2014 at 10:42 AM, Vladimir Sharshov <
> vsharshov@xxxxxxxxxxxx> wrote:
>
>> Guys, please share link to ISO. Without it i could not say anything
>> useful about potencial problem with naily. Thanks!
>>
>>
>> On Mon, Mar 24, 2014 at 4:34 PM, Andrey Danin <adanin@xxxxxxxxxxxx>wrote:
>>
>>> A huge ttl value was set in order to allow nodes with unsynchronized
>>> time be able to use mcollective. if a master node has a local time more
>>> than 6000 seconds in past in comparison with target nodes, these target
>>> nodes will not be able to answer via mcollective.
>>>
>>>
>>> On Mon, Mar 24, 2014 at 2:07 PM, Vladimir Sharshov <
>>> vsharshov@xxxxxxxxxxxx> wrote:
>>>
>>>> Hi all!
>>>>
>>>> > When I changed number of workers in naily from 3 to 2 everything
>>>> started to work
>>>> At now moment we increase this value to 7 as i remember. What about
>>>> shared connection - at now moment it works without any issues. Due to this
>>>> limitation only 2 clients in new version looks very strange.
>>>>
>>>> Please share link to iso, i try to reproduce and investigate this
>>>> problem. Thanks!
>>>>
>>>>
>>>> On Wed, Mar 19, 2014 at 1:36 PM, Mike Scherbakov <
>>>> mscherbakov@xxxxxxxxxxxx> wrote:
>>>>
>>>>> Vladimir - I think you've been working with Naily workers, any
>>>>> thoughts on the issue?
>>>>>
>>>>>
>>>>> On Wed, Mar 19, 2014 at 1:04 PM, Andrey Korolyov <
>>>>> akorolev@xxxxxxxxxxxx> wrote:
>>>>>
>>>>>> On 03/19/2014 12:52 PM, Dmitry Pyzhov wrote:
>>>>>> > + more guys.
>>>>>> >
>>>>>> >
>>>>>> > On Wed, Mar 19, 2014 at 12:10 PM, Lukasz Oles <loles@xxxxxxxxxxxx
>>>>>> > <mailto:loles@xxxxxxxxxxxx>> wrote:
>>>>>> >
>>>>>> > Hello guys,
>>>>>> >
>>>>>> > After a lot of testing and debugging finally I have something
>>>>>> to share.
>>>>>> >
>>>>>> > First change is in mcollective settings. In
>>>>>> > /etc/mcollective/server.cfg value ttl = 2000000000 is too big.
>>>>>> > Rabbitmq returns error. I changed it to 6000, without this
>>>>>> > mcollective will not work.
>>>>>>
>>>>>> The problem is not in value itself but in way how rmq drivers pushes
>>>>>> it.
>>>>>> Somehow it turns as a *concatenation* of default value around 10k and
>>>>>> this one, which is definitely too large for first one` concatenation
>>>>>> result. Just remove this value entirely as I did before from the
>>>>>> config,
>>>>>> three hours are acceptable enough.
>>>>>> >
>>>>>> > Another problem is with task status update. In my tests it
>>>>>> hangs for
>>>>>> > about 10 minutes. After that task is updated. Unfortunately
>>>>>> it's not
>>>>>> > a problem with python but with naily. When I changed number of
>>>>>> > workers in naily from 3 to 2 everything started to work. I think
>>>>>> > it's because all thread are using the same connection and
>>>>>> chanel to
>>>>>> > publish results but I'm still investigating it.
>>>>>> >
>>>>>> > Regards
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Mar 13, 2014 at 12:38 PM, Dmitry Pyzhov
>>>>>> > <dpyzhov@xxxxxxxxxxxx <mailto:dpyzhov@xxxxxxxxxxxx>> wrote:
>>>>>> >
>>>>>> > Lukasz,
>>>>>> >
>>>>>> > Feel free to contact us if you need anything else.
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Mar 13, 2014 at 3:22 PM, Lukasz Oles <
>>>>>> loles@xxxxxxxxxxxx
>>>>>> > <mailto:loles@xxxxxxxxxxxx>> wrote:
>>>>>> >
>>>>>> > ok, thx for rpm
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Mar 13, 2014 at 11:40 AM, Dmitry Burmistrov
>>>>>> > <dburmistrov@xxxxxxxxxxxx <mailto:
>>>>>> dburmistrov@xxxxxxxxxxxx>>
>>>>>> > wrote:
>>>>>> >
>>>>>> > Package rabbitmq-server has been built from
>>>>>> changeset:
>>>>>> > http://gerrit.mirantis.com/13455
>>>>>> > RPM Repository URL:
>>>>>> > http:///
>>>>>> osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos
>>>>>> > <
>>>>>> http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos
>>>>>> >
>>>>>> >
>>>>>> > Package rabbitmq-server has been built from
>>>>>> changeset:
>>>>>> > http://gerrit.mirantis.com/13457
>>>>>> > DEB Repository URL:
>>>>>> > http:///
>>>>>> osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu
>>>>>> > <
>>>>>> http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu
>>>>>> >
>>>>>> >
>>>>>> > On Thu, Mar 13, 2014 at 2:11 PM, Dmitry Pyzhov
>>>>>> > <dpyzhov@xxxxxxxxxxxx <mailto:dpyzhov@xxxxxxxxxxxx>>
>>>>>> wrote:
>>>>>> > > Lukasz,
>>>>>> > >
>>>>>> > > Sorry for long response. Our OSCI team will build
>>>>>> a
>>>>>> > package. Dmitry B, could
>>>>>> > > you reply with the download link when it is
>>>>>> available?
>>>>>> > Ticket OSCI-1016.
>>>>>> > >
>>>>>> > >
>>>>>> > > On Wed, Mar 12, 2014 at 2:18 PM, Lukasz Oles
>>>>>> > <loles@xxxxxxxxxxxx <mailto:loles@xxxxxxxxxxxx>>
>>>>>> wrote:
>>>>>> > >>
>>>>>> > >> Sure, I will look into it. Can you give me a
>>>>>> link to
>>>>>> > rabbitmq rpm which
>>>>>> > >> you used?
>>>>>> > >>
>>>>>> > >> Regards,
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> On Wed, Mar 12, 2014 at 11:10 AM, Dmitry Pyzhov
>>>>>> > <dpyzhov@xxxxxxxxxxxx <mailto:dpyzhov@xxxxxxxxxxxx
>>>>>> >>
>>>>>> > >> wrote:
>>>>>> > >>>
>>>>>> > >>> Lukasz,
>>>>>> > >>>
>>>>>> > >>> actually we have no idea what is wrong with
>>>>>> fresh
>>>>>> > rabbitmq. For some
>>>>>> > >>> reason refresh of task status takes too much
>>>>>> time.
>>>>>> > Dmitry tried to find the
>>>>>> > >>> root cause, but did not succeed. Could you
>>>>>> > investigate the issue?
>>>>>> > >>>
>>>>>> > >>>
>>>>>> > >>> On Tue, Mar 11, 2014 at 9:49 PM, Lukasz Oles
>>>>>> > <loles@xxxxxxxxxxxx <mailto:loles@xxxxxxxxxxxx>>
>>>>>> wrote:
>>>>>> > >>>>
>>>>>> > >>>> Dmitry,
>>>>>> > >>>>
>>>>>> > >>>> rabbitmq update looks interesting, I can look
>>>>>> into
>>>>>> > it. Do I need any
>>>>>> > >>>> additional information?
>>>>>> > >>>>
>>>>>> > >>>> regards,
>>>>>> > >>>>
>>>>>> > >>>>
>>>>>> > >>>> On Tue, Mar 11, 2014 at 2:05 PM, Dmitry Pyzhov
>>>>>> > <dpyzhov@xxxxxxxxxxxx <mailto:dpyzhov@xxxxxxxxxxxx
>>>>>> >>
>>>>>> > >>>> wrote:
>>>>>> > >>>>>
>>>>>> > >>>>> Great!
>>>>>> > >>>>>
>>>>>> > >>>>> Lukasz, could you help us with rabbitmq
>>>>>> update? We
>>>>>> > faced an issue with
>>>>>> > >>>>> it:
>>>>>> https://bugs.launchpad.net/fuel/+bug/1278336
>>>>>> > >>>>>
>>>>>> > >>>>> Also, could you participate it design review:
>>>>>> > >>>>>
>>>>>> >
>>>>>> https://docs.google.com/document/d/1zqV58LZBLQ-0gllb_i3MyIKIMj-Qx8ELJohjcWs459s/edit?usp=sharing
>>>>>> > >>>>>
>>>>>> > >>>>>
>>>>>> > >>>>> On Mon, Mar 10, 2014 at 9:01 PM, Mike
>>>>>> Scherbakov
>>>>>> > >>>>> <mscherbakov@xxxxxxxxxxxx
>>>>>> > <mailto:mscherbakov@xxxxxxxxxxxx>> wrote:
>>>>>> > >>>>>>
>>>>>> > >>>>>> Lukasz,
>>>>>> > >>>>>> please take any bugs from
>>>>>> > https://launchpad.net/fuel/+milestone/5.0
>>>>>> > >>>>>> which are not assigned to particular person.
>>>>>> You
>>>>>> > are likely to be interested
>>>>>> > >>>>>> in those which are assigned to
>>>>>> "fuel-python". Of
>>>>>> > course, it's preferred to
>>>>>> > >>>>>> work on Critical and High priority bugs in a
>>>>>> > first order.
>>>>>> > >>>>>>
>>>>>> > >>>>>> We are in a design phase for 5.0. Please
>>>>>> take a
>>>>>> > look at
>>>>>> > >>>>>>
>>>>>> >
>>>>>> https://mirantis.jira.com/wiki/display/PRD/5.0+-+Mirantis+OpenStack+release+home+page
>>>>>> .
>>>>>> > >>>>>> I'm discussing this still with management,
>>>>>> and we
>>>>>> > will likely have only part
>>>>>> > >>>>>> of what is on the page. Your comments and
>>>>>> input
>>>>>> > into design docs (which you
>>>>>> > >>>>>> can find following blueprint link, then
>>>>>> "Read the
>>>>>> > full spec") is very
>>>>>> > >>>>>> welcome.
>>>>>> > >>>>>>
>>>>>> > >>>>>> Start looking over and try to identify spot
>>>>>> which
>>>>>> > is in most interest
>>>>>> > >>>>>> of you. Dmitry/Evgeny will help to identify
>>>>>> areas
>>>>>> > where help is mostly
>>>>>> > >>>>>> needed. Sorry for not responding to you in
>>>>>> time.
>>>>>> > I'll get my team to fix
>>>>>> > >>>>>> this.
>>>>>> > >>>>>>
>>>>>> > >>>>>> FYI: Today is holiday in Russia & Ukraine
>>>>>> > >>>>>> Thanks,
>>>>>> > >>>>>>
>>>>>> > >>>>>>
>>>>>> > >>>>>> On Thu, Mar 6, 2014 at 12:37 AM, Mike
>>>>>> Scherbakov
>>>>>> > >>>>>> <mscherbakov@xxxxxxxxxxxx
>>>>>> > <mailto:mscherbakov@xxxxxxxxxxxx>> wrote:
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> It's great. I would be happy to see Lukasz
>>>>>> > working on Fuel.
>>>>>> > >>>>>>> Actually, Lukasz already doing great job
>>>>>> helping
>>>>>> > us with Nailgun
>>>>>> > >>>>>>> scalability issues resolution.
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> Dmitry, please arrange meeting between our
>>>>>> > Python engineers and
>>>>>> > >>>>>>> Lukasz, and identify areas where
>>>>>> contribution of
>>>>>> > Lukasz will be the most
>>>>>> > >>>>>>> effective. It should be aligned with the
>>>>>> > development of our engineers too.
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> Thanks,
>>>>>> > >>>>>>>
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> On Tue, Mar 4, 2014 at 3:27 PM, Piotr
>>>>>> Siwczak
>>>>>> > <psiwczak@xxxxxxxxxxxx <mailto:
>>>>>> psiwczak@xxxxxxxxxxxx>>
>>>>>> > >>>>>>> wrote:
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> Mike,
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> For now I see Lukasz has finished his work
>>>>>> for
>>>>>> > Softlayer/Express (at
>>>>>> > >>>>>>>> least for the next few weeks)and can use
>>>>>> his
>>>>>> > time to engage into Fuel
>>>>>> > >>>>>>>> development. Please feel free to assign
>>>>>> him to
>>>>>> > Fuel tasks.
>>>>>> > >>>>>>>>
>>>>>> > >>>>>>>> -Piotr
>>>>>> > >>>>>>>
>>>>>> > >>>>>>>
>>>>>> > >>>>>>>
>>>>>> > >>>>>>>
>>>>>> > >>>>>>> --
>>>>>> > >>>>>>> Mike Scherbakov
>>>>>> > >>>>>>> #mihgen
>>>>>> > >>>>>>
>>>>>> > >>>>>>
>>>>>> > >>>>>>
>>>>>> > >>>>>>
>>>>>> > >>>>>> --
>>>>>> > >>>>>> Mike Scherbakov
>>>>>> > >>>>>> #mihgen
>>>>>> > >>>>>
>>>>>> > >>>>>
>>>>>> > >>>>
>>>>>> > >>>>
>>>>>> > >>>>
>>>>>> > >>>> --
>>>>>> > >>>> Łukasz Oleś
>>>>>> > >>>
>>>>>> > >>>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> --
>>>>>> > >> Łukasz Oleś
>>>>>> > >
>>>>>> > >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Łukasz Oleś
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Łukasz Oleś
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > You received this message because you are subscribed to the Google
>>>>>> > Groups "fuel-core-team" group.
>>>>>> > To unsubscribe from this group and stop receiving emails from it,
>>>>>> send
>>>>>> > an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx
>>>>>> > <mailto:fuel-core-team+unsubscribe@xxxxxxxxxxxx>.
>>>>>> > For more options, visit
>>>>>> https://groups.google.com/a/mirantis.com/d/optout.
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "fuel-core-team" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
>>>>>> For more options, visit
>>>>>> https://groups.google.com/a/mirantis.com/d/optout.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mike Scherbakov
>>>>> #mihgen
>>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "fuel-core-team" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
>>>> For more options, visit
>>>> https://groups.google.com/a/mirantis.com/d/optout.
>>>>
>>>
>>>
>>>
>>> --
>>> Andrey Danin
>>> adanin@xxxxxxxxxxxx
>>> skype: gcon.monolake
>>>
>>
>>
>
>
> --
> Łukasz Oleś
>
--
Mike Scherbakov
#mihgen
Follow ups
