fuel-dev team mailing list archive
-
fuel-dev team
-
Mailing list archive
-
Message #00772
Re: Issues with RabbitMQ update
Please make sure you include a fix for this problem in fuel-library
when using RabbitMQ3 with OpenStack:
https://bugs.launchpad.net/fuel/+bug/1296922
On Wed, Apr 2, 2014 at 6:53 AM, Vladimir Sharshov
<vsharshov@xxxxxxxxxxxx> wrote:
> First of all: QA says what bug reproduced. Now i try to fix it based on
> Lukasz investigations.
>
> Short how info how to make such iso
> (http://172.18.8.252/fuelweb_rabbitmq3-5.0.iso):
>
> We need this things:
>
> 1. Patch for fuel-library (https://review.openstack.org/#/c/84707/)
> 2. Patch for fuel-main (https://review.openstack.org/#/c/84717/)
> 3. Custom RabbitMQ repo:
> http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-14055/centos
>
> Steps:
>
> http://docs.mirantis.com/fuel-dev/develop/env.html#building-the-fuel-iso
> (steps 1-4)
> cd fuel-main
> Apply patch for fuel-main:
> git fetch https://review.openstack.org/stackforge/fuel-main
> refs/changes/17/84717/1 && git cherry-pick FETCH_HEAD
> Edit config.mk:
> FUELLIB_GERRIT_COMMIT?=refs/changes/07/84707/1
> Build iso (in example i use Moscow mirror, but it is not necessary) and
> custom RabbitMQ3 package:
> make iso USE_MIRROR=msk
> EXTRA_RPM_REPOS="rabbitmq3,http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-14055/centos";
> Check iso in 'build/iso' folder
>
>
>
>
>
> On Wed, Apr 2, 2014 at 1:40 PM, Vladimir Sharshov <vsharshov@xxxxxxxxxxxx>
> wrote:
>>
>> Long awaited iso: http://172.18.8.252/fuelweb_rabbitmq3-5.0.iso (internal
>> network only)
>>
>> Using this iso we can try to repeat problem with progress bar. TTL changes
>> included.
>>
>>
>> On Mon, Mar 31, 2014 at 3:38 PM, Vladimir Sharshov
>> <vsharshov@xxxxxxxxxxxx> wrote:
>>>
>>> Hi all.
>>>
>>> Current version of RabbitMQ 3 package have issue which does not allow
>>> install it in fresh system. OSCI team now work on it. After it will be done,
>>> i will build iso for test.
>>>
>>> Sorry for delay.
>>>
>>>
>>> On Thu, Mar 27, 2014 at 4:19 PM, Vladimir Sharshov
>>> <vsharshov@xxxxxxxxxxxx> wrote:
>>>>
>>>> Thanks!
>>>>
>>>> > Do you know easy way to do it?
>>>> Yes. I do this soon.
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, Mar 27, 2014 at 3:30 PM, Lukasz Oles <loles@xxxxxxxxxxxx> wrote:
>>>>>
>>>>> hi, comments inline
>>>>>
>>>>>
>>>>> On Thu, Mar 27, 2014 at 10:50 AM, Vladimir Sharshov
>>>>> <vsharshov@xxxxxxxxxxxx> wrote:
>>>>>>
>>>>>> I try to reproduce problem with new rabbitmq from this email and all
>>>>>> works fine without any delay with 7 astute workers.
>>>>>>
>>>>>> I think we should build ISO with new RabbitMQ and ttl fixes (today i
>>>>>> spend time to reinvestigate already solved problem with ttl).
>>>>>
>>>>> Do you know easy way to do it?
>>>>>>
>>>>>>
>>>>>> Several questions for you:
>>>>>>
>>>>>> does this problem repeat only in HA-case
>>>>>> https://bugs.launchpad.net/fuel/+bug/1278336 ?
>>>>>
>>>>> No, I'm testing it by adding two nodes to cluster and pressing "Verify
>>>>> netoworks' in Netowork Tab. I'm waiting 10 min to see results on GUI.
>>>>>>
>>>>>> does problem repeat every time or episodically?
>>>>>
>>>>> Almost every time. Once it worked, with 2 astute workers, but after
>>>>> restart it stopped to work.
>>>>>>
>>>>>> does this problem affect 'generate diagnostic snapshot' (it take much
>>>>>> longer time)?
>>>>>
>>>>> I don't know. I tested it on 4.1 branch. I will repeat tests on trunk
>>>>> version.
>>>>>
>>>>>>
>>>>>> I use simple CentOS cluster (1 controller + 1 compute) and 'generate
>>>>>> diagnostic snapshot', because a few month ago in this cases UI have
>>>>>> significant delays. As result — all work without delay. Now try with HA.
>>>>>>
>>>>>>
>>>>>> On Tue, Mar 25, 2014 at 3:41 PM, Mike Scherbakov
>>>>>> <mscherbakov@xxxxxxxxxxxx> wrote:
>>>>>>>
>>>>>>> Great findings, Lukasz!
>>>>>>>
>>>>>>> Adding larger audience of fuel-dev..
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Mar 25, 2014 at 3:37 PM, Lukasz Oles <loles@xxxxxxxxxxxx>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Vladimir,
>>>>>>>>
>>>>>>>> there is no ISO, just install newest rabbitmq. I attached rpm
>>>>>>>> package for centos.
>>>>>>>>
>>>>>>>> I have done some more investigation and number of workers actually
>>>>>>>> doesn't matter. It just gives random results, but I think I found solution.
>>>>>>>>
>>>>>>>> Naily is using asynchronous library to communicate with Rabbitmq. It
>>>>>>>> uses amqp library which uses EventMachine. In Naily event loop is running in
>>>>>>>> main thread but consumer is running in another and publisher in yet another
>>>>>>>> thread.
>>>>>>>>
>>>>>>>> To solve the problem with hanging I moved the code for publisher and
>>>>>>>> consumer to EM::next_tick block. After this everything is working again now.
>>>>>>>> EM::next_tick does two things. First, it schedules code to run in
>>>>>>>> next event loop iteration. Second it runs this code in event loop thread.
>>>>>>>> I'm not sure which of this things helps.
>>>>>>>> Debugging async code in threads is really hard. Why in the first
>>>>>>>> place Naily is using async library?
>>>>>>>>
>>>>>>>> What do you think about it? Maybe it would be better just move to
>>>>>>>> synchronous library like bunny?
>>>>>>>>
>>>>>>>> Please remember I'm not ruby programmer so I can be missing
>>>>>>>> something here.
>>>>>>>>
>>>>>>>> Regards
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Mar 25, 2014 at 10:42 AM, Vladimir Sharshov
>>>>>>>> <vsharshov@xxxxxxxxxxxx> wrote:
>>>>>>>>>
>>>>>>>>> Guys, please share link to ISO. Without it i could not say anything
>>>>>>>>> useful about potencial problem with naily. Thanks!
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Mar 24, 2014 at 4:34 PM, Andrey Danin <adanin@xxxxxxxxxxxx>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> A huge ttl value was set in order to allow nodes with
>>>>>>>>>> unsynchronized time be able to use mcollective. if a master node has a local
>>>>>>>>>> time more than 6000 seconds in past in comparison with target nodes, these
>>>>>>>>>> target nodes will not be able to answer via mcollective.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Mar 24, 2014 at 2:07 PM, Vladimir Sharshov
>>>>>>>>>> <vsharshov@xxxxxxxxxxxx> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi all!
>>>>>>>>>>>
>>>>>>>>>>> > When I changed number of workers in naily from 3 to 2
>>>>>>>>>>> > everything started to work
>>>>>>>>>>> At now moment we increase this value to 7 as i remember. What
>>>>>>>>>>> about shared connection - at now moment it works without any issues. Due to
>>>>>>>>>>> this limitation only 2 clients in new version looks very strange.
>>>>>>>>>>>
>>>>>>>>>>> Please share link to iso, i try to reproduce and investigate this
>>>>>>>>>>> problem. Thanks!
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Mar 19, 2014 at 1:36 PM, Mike Scherbakov
>>>>>>>>>>> <mscherbakov@xxxxxxxxxxxx> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Vladimir - I think you've been working with Naily workers, any
>>>>>>>>>>>> thoughts on the issue?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Mar 19, 2014 at 1:04 PM, Andrey Korolyov
>>>>>>>>>>>> <akorolev@xxxxxxxxxxxx> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 03/19/2014 12:52 PM, Dmitry Pyzhov wrote:
>>>>>>>>>>>>> > + more guys.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Wed, Mar 19, 2014 at 12:10 PM, Lukasz Oles
>>>>>>>>>>>>> > <loles@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:loles@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Hello guys,
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > After a lot of testing and debugging finally I have
>>>>>>>>>>>>> > something to share.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > First change is in mcollective settings. In
>>>>>>>>>>>>> > /etc/mcollective/server.cfg value ttl = 2000000000 is
>>>>>>>>>>>>> > too big.
>>>>>>>>>>>>> > Rabbitmq returns error. I changed it to 6000, without
>>>>>>>>>>>>> > this
>>>>>>>>>>>>> > mcollective will not work.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The problem is not in value itself but in way how rmq drivers
>>>>>>>>>>>>> pushes it.
>>>>>>>>>>>>> Somehow it turns as a *concatenation* of default value around
>>>>>>>>>>>>> 10k and
>>>>>>>>>>>>> this one, which is definitely too large for first one`
>>>>>>>>>>>>> concatenation
>>>>>>>>>>>>> result. Just remove this value entirely as I did before from
>>>>>>>>>>>>> the config,
>>>>>>>>>>>>> three hours are acceptable enough.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Another problem is with task status update. In my tests
>>>>>>>>>>>>> > it hangs for
>>>>>>>>>>>>> > about 10 minutes. After that task is updated.
>>>>>>>>>>>>> > Unfortunately it's not
>>>>>>>>>>>>> > a problem with python but with naily. When I changed
>>>>>>>>>>>>> > number of
>>>>>>>>>>>>> > workers in naily from 3 to 2 everything started to work.
>>>>>>>>>>>>> > I think
>>>>>>>>>>>>> > it's because all thread are using the same connection and
>>>>>>>>>>>>> > chanel to
>>>>>>>>>>>>> > publish results but I'm still investigating it.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Regards
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Thu, Mar 13, 2014 at 12:38 PM, Dmitry Pyzhov
>>>>>>>>>>>>> > <dpyzhov@xxxxxxxxxxxx <mailto:dpyzhov@xxxxxxxxxxxx>>
>>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Lukasz,
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Feel free to contact us if you need anything else.
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Thu, Mar 13, 2014 at 3:22 PM, Lukasz Oles
>>>>>>>>>>>>> > <loles@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:loles@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > ok, thx for rpm
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Thu, Mar 13, 2014 at 11:40 AM, Dmitry
>>>>>>>>>>>>> > Burmistrov
>>>>>>>>>>>>> > <dburmistrov@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:dburmistrov@xxxxxxxxxxxx>>
>>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Package rabbitmq-server has been built from
>>>>>>>>>>>>> > changeset:
>>>>>>>>>>>>> > http://gerrit.mirantis.com/13455
>>>>>>>>>>>>> > RPM Repository URL:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > http:///osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > <http://osci-obs.vm.mirantis.net:82/centos-fuel-5.0-stable-13455/centos>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > Package rabbitmq-server has been built from
>>>>>>>>>>>>> > changeset:
>>>>>>>>>>>>> > http://gerrit.mirantis.com/13457
>>>>>>>>>>>>> > DEB Repository URL:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > http:///osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > <http://osci-obs.vm.mirantis.net:82/ubuntu-fuel-5.0-stable-13457/ubuntu>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Thu, Mar 13, 2014 at 2:11 PM, Dmitry
>>>>>>>>>>>>> > Pyzhov
>>>>>>>>>>>>> > <dpyzhov@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:dpyzhov@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>>>> > > Lukasz,
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > Sorry for long response. Our OSCI team will
>>>>>>>>>>>>> > build a
>>>>>>>>>>>>> > package. Dmitry B, could
>>>>>>>>>>>>> > > you reply with the download link when it is
>>>>>>>>>>>>> > available?
>>>>>>>>>>>>> > Ticket OSCI-1016.
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > On Wed, Mar 12, 2014 at 2:18 PM, Lukasz
>>>>>>>>>>>>> > Oles
>>>>>>>>>>>>> > <loles@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:loles@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >> Sure, I will look into it. Can you give me
>>>>>>>>>>>>> > a link to
>>>>>>>>>>>>> > rabbitmq rpm which
>>>>>>>>>>>>> > >> you used?
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >> Regards,
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >> On Wed, Mar 12, 2014 at 11:10 AM, Dmitry
>>>>>>>>>>>>> > Pyzhov
>>>>>>>>>>>>> > <dpyzhov@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:dpyzhov@xxxxxxxxxxxx>>
>>>>>>>>>>>>> > >> wrote:
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> > >>> Lukasz,
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> > >>> actually we have no idea what is wrong
>>>>>>>>>>>>> > with fresh
>>>>>>>>>>>>> > rabbitmq. For some
>>>>>>>>>>>>> > >>> reason refresh of task status takes too
>>>>>>>>>>>>> > much time.
>>>>>>>>>>>>> > Dmitry tried to find the
>>>>>>>>>>>>> > >>> root cause, but did not succeed. Could
>>>>>>>>>>>>> > you
>>>>>>>>>>>>> > investigate the issue?
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> > >>> On Tue, Mar 11, 2014 at 9:49 PM, Lukasz
>>>>>>>>>>>>> > Oles
>>>>>>>>>>>>> > <loles@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:loles@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> Dmitry,
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> rabbitmq update looks interesting, I can
>>>>>>>>>>>>> > look into
>>>>>>>>>>>>> > it. Do I need any
>>>>>>>>>>>>> > >>>> additional information?
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> regards,
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> On Tue, Mar 11, 2014 at 2:05 PM, Dmitry
>>>>>>>>>>>>> > Pyzhov
>>>>>>>>>>>>> > <dpyzhov@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:dpyzhov@xxxxxxxxxxxx>>
>>>>>>>>>>>>> > >>>> wrote:
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>> Great!
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>> Lukasz, could you help us with rabbitmq
>>>>>>>>>>>>> > update? We
>>>>>>>>>>>>> > faced an issue with
>>>>>>>>>>>>> > >>>>> it:
>>>>>>>>>>>>> > https://bugs.launchpad.net/fuel/+bug/1278336
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>> Also, could you participate it design
>>>>>>>>>>>>> > review:
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > https://docs.google.com/document/d/1zqV58LZBLQ-0gllb_i3MyIKIMj-Qx8ELJohjcWs459s/edit?usp=sharing
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>> On Mon, Mar 10, 2014 at 9:01 PM, Mike
>>>>>>>>>>>>> > Scherbakov
>>>>>>>>>>>>> > >>>>> <mscherbakov@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:mscherbakov@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> Lukasz,
>>>>>>>>>>>>> > >>>>>> please take any bugs from
>>>>>>>>>>>>> > https://launchpad.net/fuel/+milestone/5.0
>>>>>>>>>>>>> > >>>>>> which are not assigned to particular
>>>>>>>>>>>>> > person. You
>>>>>>>>>>>>> > are likely to be interested
>>>>>>>>>>>>> > >>>>>> in those which are assigned to
>>>>>>>>>>>>> > "fuel-python". Of
>>>>>>>>>>>>> > course, it's preferred to
>>>>>>>>>>>>> > >>>>>> work on Critical and High priority
>>>>>>>>>>>>> > bugs in a
>>>>>>>>>>>>> > first order.
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> We are in a design phase for 5.0.
>>>>>>>>>>>>> > Please take a
>>>>>>>>>>>>> > look at
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > https://mirantis.jira.com/wiki/display/PRD/5.0+-+Mirantis+OpenStack+release+home+page.
>>>>>>>>>>>>> > >>>>>> I'm discussing this still with
>>>>>>>>>>>>> > management, and we
>>>>>>>>>>>>> > will likely have only part
>>>>>>>>>>>>> > >>>>>> of what is on the page. Your comments
>>>>>>>>>>>>> > and input
>>>>>>>>>>>>> > into design docs (which you
>>>>>>>>>>>>> > >>>>>> can find following blueprint link,
>>>>>>>>>>>>> > then "Read the
>>>>>>>>>>>>> > full spec") is very
>>>>>>>>>>>>> > >>>>>> welcome.
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> Start looking over and try to identify
>>>>>>>>>>>>> > spot which
>>>>>>>>>>>>> > is in most interest
>>>>>>>>>>>>> > >>>>>> of you. Dmitry/Evgeny will help to
>>>>>>>>>>>>> > identify areas
>>>>>>>>>>>>> > where help is mostly
>>>>>>>>>>>>> > >>>>>> needed. Sorry for not responding to
>>>>>>>>>>>>> > you in time.
>>>>>>>>>>>>> > I'll get my team to fix
>>>>>>>>>>>>> > >>>>>> this.
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> FYI: Today is holiday in Russia &
>>>>>>>>>>>>> > Ukraine
>>>>>>>>>>>>> > >>>>>> Thanks,
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> On Thu, Mar 6, 2014 at 12:37 AM, Mike
>>>>>>>>>>>>> > Scherbakov
>>>>>>>>>>>>> > >>>>>> <mscherbakov@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:mscherbakov@xxxxxxxxxxxx>> wrote:
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>> It's great. I would be happy to see
>>>>>>>>>>>>> > Lukasz
>>>>>>>>>>>>> > working on Fuel.
>>>>>>>>>>>>> > >>>>>>> Actually, Lukasz already doing great
>>>>>>>>>>>>> > job helping
>>>>>>>>>>>>> > us with Nailgun
>>>>>>>>>>>>> > >>>>>>> scalability issues resolution.
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>> Dmitry, please arrange meeting
>>>>>>>>>>>>> > between our
>>>>>>>>>>>>> > Python engineers and
>>>>>>>>>>>>> > >>>>>>> Lukasz, and identify areas where
>>>>>>>>>>>>> > contribution of
>>>>>>>>>>>>> > Lukasz will be the most
>>>>>>>>>>>>> > >>>>>>> effective. It should be aligned with
>>>>>>>>>>>>> > the
>>>>>>>>>>>>> > development of our engineers too.
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>> Thanks,
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>> On Tue, Mar 4, 2014 at 3:27 PM, Piotr
>>>>>>>>>>>>> > Siwczak
>>>>>>>>>>>>> > <psiwczak@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:psiwczak@xxxxxxxxxxxx>>
>>>>>>>>>>>>> > >>>>>>> wrote:
>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>> > >>>>>>>> Mike,
>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>> > >>>>>>>> For now I see Lukasz has finished
>>>>>>>>>>>>> > his work for
>>>>>>>>>>>>> > Softlayer/Express (at
>>>>>>>>>>>>> > >>>>>>>> least for the next few weeks)and can
>>>>>>>>>>>>> > use his
>>>>>>>>>>>>> > time to engage into Fuel
>>>>>>>>>>>>> > >>>>>>>> development. Please feel free to
>>>>>>>>>>>>> > assign him to
>>>>>>>>>>>>> > Fuel tasks.
>>>>>>>>>>>>> > >>>>>>>>
>>>>>>>>>>>>> > >>>>>>>> -Piotr
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>>
>>>>>>>>>>>>> > >>>>>>> --
>>>>>>>>>>>>> > >>>>>>> Mike Scherbakov
>>>>>>>>>>>>> > >>>>>>> #mihgen
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>>
>>>>>>>>>>>>> > >>>>>> --
>>>>>>>>>>>>> > >>>>>> Mike Scherbakov
>>>>>>>>>>>>> > >>>>>> #mihgen
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>>
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>>
>>>>>>>>>>>>> > >>>> --
>>>>>>>>>>>>> > >>>> Łukasz Oleś
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> > >>>
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >>
>>>>>>>>>>>>> > >> --
>>>>>>>>>>>>> > >> Łukasz Oleś
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > --
>>>>>>>>>>>>> > Łukasz Oleś
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > --
>>>>>>>>>>>>> > Łukasz Oleś
>>>>>>>>>>>>> >
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > --
>>>>>>>>>>>>> > You received this message because you are subscribed to the
>>>>>>>>>>>>> > Google
>>>>>>>>>>>>> > Groups "fuel-core-team" group.
>>>>>>>>>>>>> > To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>>> > it, send
>>>>>>>>>>>>> > an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx
>>>>>>>>>>>>> > <mailto:fuel-core-team+unsubscribe@xxxxxxxxxxxx>.
>>>>>>>>>>>>> > For more options, visit
>>>>>>>>>>>>> > https://groups.google.com/a/mirantis.com/d/optout.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>>>> Google Groups "fuel-core-team" group.
>>>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from
>>>>>>>>>>>>> it, send an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
>>>>>>>>>>>>> For more options, visit
>>>>>>>>>>>>> https://groups.google.com/a/mirantis.com/d/optout.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Mike Scherbakov
>>>>>>>>>>>> #mihgen
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> You received this message because you are subscribed to the
>>>>>>>>>>> Google Groups "fuel-core-team" group.
>>>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>>>> send an email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
>>>>>>>>>>> For more options, visit
>>>>>>>>>>> https://groups.google.com/a/mirantis.com/d/optout.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Andrey Danin
>>>>>>>>>> adanin@xxxxxxxxxxxx
>>>>>>>>>> skype: gcon.monolake
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Łukasz Oleś
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Mike Scherbakov
>>>>>>> #mihgen
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Łukasz Oleś
>>>>
>>>>
>>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "fuel-core-team" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to fuel-core-team+unsubscribe@xxxxxxxxxxxx.
> For more options, visit https://groups.google.com/a/mirantis.com/d/optout.
--
Dmitry Borodaenko
Follow ups
References
