fuel-dev team mailing list archive
-
fuel-dev team
-
Mailing list archive
-
Message #00763
Re: Issues with RabbitMQ update
Long awaited iso: http://172.18.8.252/fuelweb_rabbitmq3-5.0.iso (internal
network only)
Using this iso we can try to repeat problem with progress bar. TTL changes
included.
On Mon, Mar 31, 2014 at 3:38 PM, Vladimir Sharshov
<vsharshov@xxxxxxxxxxxx>wrote:
> Hi all.
>
> Current version of RabbitMQ 3 package have issue which does not allow
> install it in fresh system. OSCI team now work on it. After it will be
> done, i will build iso for test.
>
> Sorry for delay.
>
>
> On Thu, Mar 27, 2014 at 4:19 PM, Vladimir Sharshov <vsharshov@xxxxxxxxxxxx
> > wrote:
>
>> Thanks!
>>
>> > Do you know easy way to do it?
>> Yes. I do this soon.
>>
>>
>>
>>
>> On Thu, Mar 27, 2014 at 3:30 PM, Lukasz Oles <loles@xxxxxxxxxxxx> wrote:
>>
>>> hi, comments inline
>>>
>>>
>>> On Thu, Mar 27, 2014 at 10:50 AM, Vladimir Sharshov <
>>> vsharshov@xxxxxxxxxxxx> wrote:
>>>>
>>>> I try to reproduce problem with new rabbitmq from this email and all
>>>> works fine without any delay with 7 astute workers.
>>>>
>>>> I think we should build ISO with new RabbitMQ and ttl fixes (today i
>>>> spend time to reinvestigate already solved problem with ttl).
>>>>
>>> Do you know easy way to do it?
>>>
>>>>
>>>> Several questions for you:
>>>>
>>>> - does this problem repeat only in HA-case
>>>> https://bugs.launchpad.net/fuel/+bug/1278336 ?
>>>>
>>>> No, I'm testing it by adding two nodes to cluster and pressing "Verify
>>> netoworks' in Netowork Tab. I'm waiting 10 min to see results on GUI.
>>>
>>>>
>>>> - does problem repeat every time or episodically?
>>>>
>>>> Almost every time. Once it worked, with 2 astute workers, but after
>>> restart it stopped to work.
>>>
>>>>
>>>> - does this problem affect 'generate diagnostic snapshot' (it take
>>>> much longer time)?
>>>>
>>>> I don't know. I tested it on 4.1 branch. I will repeat tests on trunk
>>> version.
>>>
>>>
>>>> I use simple CentOS cluster (1 controller + 1 compute) and 'generate
>>>> diagnostic snapshot', because a few month ago in this cases UI
>>>> have significant delays. As result — all work without delay. Now try with
>>>> HA.
>>>>
>>>>
>>>> On Tue, Mar 25, 2014 at 3:41 PM, Mike Scherbakov <
>>>> mscherbakov@xxxxxxxxxxxx> wrote:
>>>>
>>>>> Great findings, Lukasz!
>>>>>
>>>>> Adding larger audience of fuel-dev..
>>>>>
>>>>>
>>>>> On Tue, Mar 25, 2014 at 3:37 PM, Lukasz Oles <loles@xxxxxxxxxxxx>wrote:
>>>>>
>>>>>> Vladimir,
>>>>>>
>>>>>> there is no ISO, just install newest rabbitmq. I attached rpm package
>>>>>> for centos.
>>>>>>
>>>>>> I have done some more investigation and number of workers actually
>>>>>> doesn't matter. It just gives random results, but I think I found solution.
>>>>>>
>>>>>> Naily is using asynchronous library to communicate with Rabbitmq. It
>>>>>> uses amqp library which uses EventMachine. In Naily event loop is running
>>>>>> in main thread but consumer is running in another and publisher in yet
>>>>>> another thread.
>>>>>>
>>>>>> To solve the problem with hanging I moved the code for publisher and
>>>>>> consumer to EM::next_tick block. After this everything is working again
>>>>>> now.
>>>>>> EM::next_tick does two things. First, it schedules code to run in
>>>>>> next event loop iteration. Second it runs this code in event loop thread.
>>>>>> I'm not sure which of this things helps.
>>>>>> Debugging async code in threads is really hard. Why in the first
>>>>>> place Naily is using async library?
>>>>>>
>>>>>> What do you think about it? Maybe it would be better just move to
>>>>>> synchronous library like bunny?
>>>>>>
>>>>>> Please remember I'm not ruby programmer so I can be missing something
>>>>>> here.
>>>>>>
>>>>>> Regards
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 25, 2014 at 10:42 AM, Vladimir Sharshov <
>>>>>> vsharshov@xxxxxxxxxxxx> wrote:
>>>>>>
>>>>>>> Guys, please share link to ISO. Without it i could not say anything
>>>>>>> useful about potencial problem with naily. Thanks!
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Mar 24, 2014 at 4:34 PM, Andrey Danin <adanin@xxxxxxxxxxxx>wrote:
>>>>>>>
>>>>>>>> A huge ttl value was set in order to allow nodes with
>>>>>>>> unsynchronized time be able to use mcollective. if a master node has a
>>>>>>>> local time more than 6000 seconds in past in comparison with target nodes,
>>>>>>>> these target nodes will not be able to answer via mcollective.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 24, 2014 at 2:07 PM, Vladimir Sharshov <
>>>>>>>> vsharshov@xxxxxxxxxxxx> wrote:
>>>>>>>>
>>>>>>>>> Hi all!
>>>>>>>>>
>>>>>>>>> > When I changed number of workers in naily from 3 to 2 everything
>>>>>>>>> started to work
>>>>>>>>> At now moment we increase this value to 7 as i remember. What
>>>>>>>>> about shared connection - at now moment it works without any issues. Due to
>>>>>>>>> this limitation only 2 clients in new version looks very strange.
>>>>>>>>>
>>>>>>>>> Please share link to iso, i try to reproduce and investigate this
>>>>>>>>> problem. Thanks!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Mar 19, 2014 at 1:36 PM, Mike Scherbakov <
>>>>>>>>> mscherbakov@xxxxxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>>> Vladimir - I think you've been working with Naily workers, any
>>>>>>>>>> thoughts on the issue?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Mar 19, 2014 at 1:04 PM, Andrey Korolyov <
>>>>>>>>>> akorolev@xxxxxxxxxxxx> wrote:
>>>>>>>>>>
>>>>>>>>>>> On 03/19/2014 12:52 PM, Dmitry Pyzhov wrote:
>>>>>>>>>>> > + more guys.
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Wed, Mar 19, 2014 at 12:10 PM, Lukasz Oles <
>>>>>>>>>>> loles@xxxxxxxxxxxx
>>>>>>>>>>> > <mailto:loles@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > Hello guys,
>>>>>>>>>>> >
>>>>>>>>>>> > After a lot of testing and debugging finally I have
>>>>>>>>>>> something to share.
>>>>>>>>>>> >
>>>>>>>>>>> > First change is in mcollective settings. In
>>>>>>>>>>> > /etc/mcollective/server.cfg value ttl = 2000000000 is too
>>>>>>>>>>> big.
>>>>>>>>>>> > Rabbitmq returns error. I changed it to 6000, without this
>>>>>>>>>>> > mcollective will not work.
>>>>>>>>>>>
>>>>>>>>>>> The problem is not in value itself but in way how rmq drivers
>>>>>>>>>>> pushes it.
>>>>>>>>>>> Somehow it turns as a *concatenation* of default value around
>>>>>>>>>>> 10k and
>>>>>>>>>>> this one, which is definitely too large for first one`
>>>>>>>>>>> concatenation
>>>>>>>>>>> result. Just remove this value entirely as I did before from the
>>>>>>>>>>> config,
>>>>>>>>>>> three hours are acceptable enough.
>>>>>>>>>>> >
>>>>>>>>>>> > Another problem is with task status update. In my tests it
>>>>>>>>>>> hangs for
>>>>>>>>>>> > about 10 minutes. After that task is updated.
>>>>>>>>>>> Unfortunately it's not
>>>>>>>>>>> > a problem with python but with naily. When I changed
>>>>>>>>>>> number of
>>>>>>>>>>> > workers in naily from 3 to 2 everything started to work. I
>>>>>>>>>>> think
>>>>>>>>>>> > it's because all thread are using the same connection and
>>>>>>>>>>> chanel to
>>>>>>>>>>> > publish results but I'm still investigating it.
>>>>>>>>>>> >
>>>>>>>>>>> > Regards
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Thu, Mar 13, 2014 at 12:38 PM, Dmitry Pyzhov
>>>>>>>>>>> > <dpyzhov@xxxxxxxxxxxx <mailto:dpyzhov@xxxxxxxxxxxx>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > Lukasz,
>>>>>>>>>>> >
>>>>>>>>>>> > Feel free to contact us if you need anything else.
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Thu, Mar 13, 2014 at 3:22 PM, Lukasz Oles <
>>>>>>>>>>> loles@xxxxxxxxxxxx
>>>>>>>>>>> > <mailto:loles@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > ok, thx for rpm
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Thu, Mar 13, 2014 at 11:40 AM, Dmitry Burmistrov
>>>>>>>>>>> > <dburmistrov@xxxxxxxxxxxx <mailto:
>>>>>>>>>>> dburmistrov@xxxxxxxxxxxx>>
>>>>>>>>>>> > wrote:
>>>>>>>>>>> >
>>>>>>>>>>> > Package rabbitmq-server has been built from
>>>>>>>>>>> changeset:
>>>>>>>>>>> > http://gerrit.mirantis.com/13455
>>>>>>>>>>> > RPM Repository URL:
>>>>>>>>>>> > http:///
>>>>>>>>>>> osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos
>>>>>>>>>>> > <
>>>>>>>>>>> http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > Package rabbitmq-server has been built from
>>>>>>>>>>> changeset:
>>>>>>>>>>> > http://gerrit.mirantis.com/13457
>>>>>>>>>>> > DEB Repository URL:
>>>>>>>>>>> > http:///
>>>>>>>>>>> osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu
>>>>>>>>>>> > <
>>>>>>>>>>> http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > On Thu, Mar 13, 2014 at 2:11 PM, Dmitry Pyzhov
>>>>>>>>>>> > <dpyzhov@xxxxxxxxxxxx <mailto:
>>>>>>>>>>> dpyzhov@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>> > > Lukasz,
>>>>>>>>>>> > >
>>>>>>>>>>> > > Sorry for long response. Our OSCI team will
>>>>>>>>>>> build a
>>>>>>>>>>> > package. Dmitry B, could
>>>>>>>>>>> > > you reply with the download link when it is
>>>>>>>>>>> available?
>>>>>>>>>>> > Ticket OSCI-1016.
>>>>>>>>>>> > >
>>>>>>>>>>> > >
>>>>>>>>>>> > > On Wed, Mar 12, 2014 at 2:18 PM, Lukasz Oles
>>>>>>>>>>> > <loles@xxxxxxxxxxxx <mailto:loles@xxxxxxxxxxxx>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Sure, I will look into it. Can you give me
>>>>>>>>>>> a link to
>>>>>>>>>>> > rabbitmq rpm which
>>>>>>>>>>> > >> you used?
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> Regards,
>>>>>>>>>>> > >>
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> On Wed, Mar 12, 2014 at 11:10 AM, Dmitry
>>>>>>>>>>> Pyzhov
>>>>>>>>>>> > <dpyzhov@xxxxxxxxxxxx <mailto:
>>>>>>>>>>> dpyzhov@xxxxxxxxxxxx>>
>>>>>>>>>>> > >> wrote:
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>> Lukasz,
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>> actually we have no idea what is wrong
>>>>>>>>>>> with fresh
>>>>>>>>>>> > rabbitmq. For some
>>>>>>>>>>> > >>> reason refresh of task status takes too
>>>>>>>>>>> much time.
>>>>>>>>>>> > Dmitry tried to find the
>>>>>>>>>>> > >>> root cause, but did not succeed. Could you
>>>>>>>>>>> > investigate the issue?
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>> On Tue, Mar 11, 2014 at 9:49 PM, Lukasz
>>>>>>>>>>> Oles
>>>>>>>>>>> > <loles@xxxxxxxxxxxx <mailto:loles@xxxxxxxxxxxx>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> Dmitry,
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> rabbitmq update looks interesting, I can
>>>>>>>>>>> look into
>>>>>>>>>>> > it. Do I need any
>>>>>>>>>>> > >>>> additional information?
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> regards,
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> On Tue, Mar 11, 2014 at 2:05 PM, Dmitry
>>>>>>>>>>> Pyzhov
>>>>>>>>>>> > <dpyzhov@xxxxxxxxxxxx <mailto:
>>>>>>>>>>> dpyzhov@xxxxxxxxxxxx>>
>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>> Great!
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>> Lukasz, could you help us with rabbitmq
>>>>>>>>>>> update? We
>>>>>>>>>>> > faced an issue with
>>>>>>>>>>> > >>>>> it:
>>>>>>>>>>> https://bugs.launchpad.net/fuel/+bug/1278336
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>> Also, could you participate it design
>>>>>>>>>>> review:
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> >
>>>>>>>>>>> https://docs.google.com/document/d/1zqV58LZBLQ-0gllb_i3MyIKIMj-Qx8ELJohjcWs459s/edit?usp=sharing
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>> On Mon, Mar 10, 2014 at 9:01 PM, Mike
>>>>>>>>>>> Scherbakov
>>>>>>>>>>> > >>>>> <mscherbakov@xxxxxxxxxxxx
>>>>>>>>>>> > <mailto:mscherbakov@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> Lukasz,
>>>>>>>>>>> > >>>>>> please take any bugs from
>>>>>>>>>>> > https://launchpad.net/fuel/+milestone/5.0
>>>>>>>>>>> > >>>>>> which are not assigned to particular
>>>>>>>>>>> person. You
>>>>>>>>>>> > are likely to be interested
>>>>>>>>>>> > >>>>>> in those which are assigned to
>>>>>>>>>>> "fuel-python". Of
>>>>>>>>>>> > course, it's preferred to
>>>>>>>>>>> > >>>>>> work on Critical and High priority bugs
>>>>>>>>>>> in a
>>>>>>>>>>> > first order.
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> We are in a design phase for 5.0.
>>>>>>>>>>> Please take a
>>>>>>>>>>> > look at
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> >
>>>>>>>>>>> https://mirantis.jira.com/wiki/display/PRD/5.0+-+Mirantis+OpenStack+release+home+page
>>>>>>>>>>> .
>>>>>>>>>>> > >>>>>> I'm discussing this still with
>>>>>>>>>>> management, and we
>>>>>>>>>>> > will likely have only part
>>>>>>>>>>> > >>>>>> of what is on the page. Your comments
>>>>>>>>>>> and input
>>>>>>>>>>> > into design docs (which you
>>>>>>>>>>> > >>>>>> can find following blueprint link, then
>>>>>>>>>>> "Read the
>>>>>>>>>>> > full spec") is very
>>>>>>>>>>> > >>>>>> welcome.
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> Start looking over and try to identify
>>>>>>>>>>> spot which
>>>>>>>>>>> > is in most interest
>>>>>>>>>>> > >>>>>> of you. Dmitry/Evgeny will help to
>>>>>>>>>>> identify areas
>>>>>>>>>>> > where help is mostly
>>>>>>>>>>> > >>>>>> needed. Sorry for not responding to you
>>>>>>>>>>> in time.
>>>>>>>>>>> > I'll get my team to fix
>>>>>>>>>>> > >>>>>> this.
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> FYI: Today is holiday in Russia &
>>>>>>>>>>> Ukraine
>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> On Thu, Mar 6, 2014 at 12:37 AM, Mike
>>>>>>>>>>> Scherbakov
>>>>>>>>>>> > >>>>>> <mscherbakov@xxxxxxxxxxxx
>>>>>>>>>>> > <mailto:mscherbakov@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> It's great. I would be happy to see
>>>>>>>>>>> Lukasz
>>>>>>>>>>> > working on Fuel.
>>>>>>>>>>> > >>>>>>> Actually, Lukasz already doing great
>>>>>>>>>>> job helping
>>>>>>>>>>> > us with Nailgun
>>>>>>>>>>> > >>>>>>> scalability issues resolution.
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> Dmitry, please arrange meeting between
>>>>>>>>>>> our
>>>>>>>>>>> > Python engineers and
>>>>>>>>>>> > >>>>>>> Lukasz, and identify areas where
>>>>>>>>>>> contribution of
>>>>>>>>>>> > Lukasz will be the most
>>>>>>>>>>> > >>>>>>> effective. It should be aligned with
>>>>>>>>>>> the
>>>>>>>>>>> > development of our engineers too.
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> Thanks,
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> On Tue, Mar 4, 2014 at 3:27 PM, Piotr
>>>>>>>>>>> Siwczak
>>>>>>>>>>> > <psiwczak@xxxxxxxxxxxx <mailto:
>>>>>>>>>>> psiwczak@xxxxxxxxxxxx>>
>>>>>>>>>>> > >>>>>>> wrote:
>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>> > >>>>>>>> Mike,
>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>> > >>>>>>>> For now I see Lukasz has finished his
>>>>>>>>>>> work for
>>>>>>>>>>> > Softlayer/Express (at
>>>>>>>>>>> > >>>>>>>> least for the next few weeks)and can
>>>>>>>>>>> use his
>>>>>>>>>>> > time to engage into Fuel
>>>>>>>>>>> > >>>>>>>> development. Please feel free to
>>>>>>>>>>> assign him to
>>>>>>>>>>> > Fuel tasks.
>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>> > >>>>>>>> -Piotr
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>> > >>>>>>> --
>>>>>>>>>>> > >>>>>>> Mike Scherbakov
>>>>>>>>>>> > >>>>>>> #mihgen
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>>
>>>>>>>>>>> > >>>>>> --
>>>>>>>>>>> > >>>>>> Mike Scherbakov
>>>>>>>>>>> > >>>>>> #mihgen
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>>
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>>
>>>>>>>>>>> > >>>> --
>>>>>>>>>>> > >>>> Łukasz Oleś
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>>
>>>>>>>>>>> > >>
>>>>>>>>>>> > >>
>>>>>>>>>>> > >>
>>>>>>>>>>> > >> --
>>>>>>>>>>> > >> Łukasz Oleś
>>>>>>>>>>> > >
>>>>>>>>>>> > >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > --
>>>>>>>>>>> > Łukasz Oleś
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > --
>>>>>>>>>>> > Łukasz Oleś
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > --
>>>>>>>>>>> > You received this message because you are subscribed to the
>>>>>>>>>>> Google
>>>>>>>>>>> > Groups "fuel-core-team" group.
>>>>>>>>>>> > To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send
>>>>>>>>>>> > an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx
>>>>>>>>>>> > <mailto:fuel-core-team+unsubscribe@xxxxxxxxxxxx>.
>>>>>>>>>>> > For more options, visit
>>>>>>>>>>> https://groups.google.com/a/mirantis.com/d/optout.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "fuel-core-team" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>> it, send an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
>>>>>>>>>>> For more options, visit
>>>>>>>>>>> https://groups.google.com/a/mirantis.com/d/optout.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Mike Scherbakov
>>>>>>>>>> #mihgen
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "fuel-core-team" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
>>>>>>>>> For more options, visit
>>>>>>>>> https://groups.google.com/a/mirantis.com/d/optout.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Andrey Danin
>>>>>>>> adanin@xxxxxxxxxxxx
>>>>>>>> skype: gcon.monolake
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Łukasz Oleś
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mike Scherbakov
>>>>> #mihgen
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Łukasz Oleś
>>>
>>
>>
>
Follow ups
References
