fuel-dev team mailing list archive

[Dmitry Nikishov <nikishov.da@xxxxxxxxx>]

Alright, there's the first part of it: server installation.
https://review.openstack.org/#/c/79566/

Would appreciate any feedback on it.


2014-03-07 14:40 GMT+04:00 Vladimir Kuklin <vkuklin@xxxxxxxxxxxx>:

> Dmitry
>
> It would be awesome if you could split the request into several ones. E.
> g. one for server installation. One for each openstack service template.
> One for adding templates to the hosts. Thus, it will allow us to merge it
> smoothly without a headache.
> 07 марта 2014 г. 14:21 пользователь "Dmitry Nikishov" <
> nikishov.da@xxxxxxxxx> написал:
>
> There's a basic version of my own reimplementation available for review:
>> https://review.openstack.org/#/c/78919/
>> Configuration templates were taken from PL team's module with their
>> permission.
>> It's able to deploy zabbix server and configure zabbix agents to monitor
>> openstack services. It still requires lots of work so it should not be
>> merged, but it would be great to get some feedback on it.
>>
>>
>> 2014-02-21 14:55 GMT+04:00 Roman Zhnichkov <rzhnichkov@xxxxxxxxxxxx>:
>>
>>> Andrew,
>>>   thanks for this catch, we'll investigate alternatives.
>>>
>>> --
>>> Roman Zhnichkov
>>>
>>>
>>>
>>> On Thu, Feb 20, 2014 at 11:00 PM, Andrew Woodward <xarses@xxxxxxxxx>wrote:
>>>
>>>> I've reviewed most of the zabbix commit request and I see that it
>>>> contains many uses of stored configs which wont work any longer since
>>>> we don't have the puppet db / master anymore so there is likely a bit
>>>> of work there. Additionally we should not merge it because it's
>>>> licensed under AGPL which at this point will not be accepted. I've
>>>> reached out to the upstream authors and have requested a change in
>>>> license but this should not be expected. We should be reviewing
>>>> additional module providers that have a more suitable upstream license
>>>> and possibly less work to work in our environment.
>>>>
>>>> On Wed, Feb 19, 2014 at 11:34 AM, Dmitry Borodaenko
>>>> <dborodaenko@xxxxxxxxxxxx> wrote:
>>>> > David,
>>>> >
>>>> > As far as I understand the difference between #4 and #1 is that in the
>>>> > case of #4 Zabbix server is not deployed by Fuel (pre-existing
>>>> > monitoring server), so our job is to collect the server details from
>>>> > the operator and configure Zabbix agents to report to that server.
>>>> >
>>>> > -DmitryB
>>>> >
>>>> > On Wed, Feb 19, 2014 at 6:56 AM, David Easter <deaster@xxxxxxxxxxxx>
>>>> wrote:
>>>> >> Use cases 1 & 2 are fine for MVP in the first release, yes.
>>>> >>
>>>> >> Would the mechanism for deploying use case #4 be the same as use
>>>> case #1 -
>>>> >> just configuring the server to monitor agents in multiple clouds?
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >> - David J. Easter
>>>> >>   Product Line Manager,  Mirantis
>>>> >>
>>>> >> From: Roman Zhnichkov <rzhnichkov@xxxxxxxxxxxx>
>>>> >> Date: Wednesday, February 19, 2014 at 4:30 AM
>>>> >> To: David Easter <deaster@xxxxxxxxxxxx>
>>>> >> Cc: "fuel-dev@xxxxxxxxxxxxxxxxxxx" <fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>> >> Subject: Re: [Fuel-dev] Zabbix feature review request
>>>> >>
>>>> >> Guys,
>>>> >>   in other words there are several use cases for Zabbix server
>>>> placement:
>>>> >>
>>>> >> Zabbix server as separate role on dedicated hardware
>>>> >> Zabbix server role combined with some other role (compute or
>>>> storage, since
>>>> >> Zabbix uses MySQL, so we don't want to mix OpenStack and Zabbix data
>>>> in the
>>>> >> same DB)
>>>> >> Zabbix server on fuel-master
>>>> >> Remote Zabbix server outside of OpenStack environment
>>>> >> Combination of any of 4 previous items (for example, we have Zabbix
>>>> server
>>>> >> on fuel-master monitoring all existing environments and some other
>>>> Zabbix
>>>> >> servers monitoring particular OpenStack environments)
>>>> >>
>>>> >> Also we have general requirements:
>>>> >>
>>>> >> HA for Zabbix server
>>>> >> Dynamic configuration (automatic node deletion when removing node
>>>> from the
>>>> >> environment, requires orchestration patching)
>>>> >>
>>>> >> As MVP I propose to have use cases 1 and 2 implemented (Zabbix
>>>> server as
>>>> >> separate role ans Zabbix server role combined with compute or
>>>> storage) with
>>>> >> dynamic configuration requirement.
>>>> >>
>>>> >> Use cases 3-5 as well as HA for Zabbix are subject to further
>>>> >> implementation.
>>>> >>
>>>> >> We'll update the specification according with that.
>>>> >>
>>>> >> Any comments or suggestions?
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Roman Zhnichkov
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Tue, Feb 18, 2014 at 8:21 PM, David Easter <deaster@xxxxxxxxxxxx>
>>>> wrote:
>>>> >>>
>>>> >>> On 2/18/14, 4:23 AM, "Bogdan Dobrelya" <bdobrelia@xxxxxxxxxxxx>
>>>> wrote:
>>>> >>>
>>>> >>> >On 02/18/2014 12:05 PM, Dmitry Nikishov wrote:
>>>> >>> >> From this conversation we got following points:
>>>> >>> >> 1. It should be possible to install zabbix as a separate role
>>>> >>> >> 2. Combining zabbix role with other roles should be supported as
>>>> well
>>>> >>> >> 3. There should be support for configuring zabbix agents to
>>>> point them
>>>> >>> >> to the remote zabbix server
>>>> >>> >> 4. Possible zabbix installation on fuel-master?
>>>> >>> >> 5. In case of one zabbix server monitoring multiple
>>>> environments, nodes
>>>> >>> >> should be separated into hostgroups.
>>>> >>> >>
>>>> >>> >> Concerns:
>>>> >>> >> 1. perfomance when alot of nodes are being monitored by single
>>>> server
>>>> >>>
>>>> >>> True.  We'd need to publish the maximum number of nodes that can be
>>>> >>> monitored efficiently by one server.  If the number of monitored
>>>> nodes
>>>> >>> goes above that number, we'd recommend a dedicated server within the
>>>> >>> largest deployed cloud (for example) to split the burden.
>>>> >>>
>>>> >>> >> 2. if zabbix agents were configured to connect to the remote
>>>> server
>>>> >>> >> (outside the environment), once this env gets deleted, it's
>>>> nodes will
>>>> >>> >> be rebooted into bootstrap. However, they won't be deleted from
>>>> zabbix.
>>>> >>> >> It's not clear how to remove them from zabbix server.
>>>> >>>
>>>> >>> We'd have to build in automation to do that for the user.  I.e. if a
>>>> >>> remote node is deleted, update the corresponding zabbix server's
>>>> info.
>>>> >>>
>>>> >>> >>
>>>> >>> >> Open question: if there will be support for zabbix on
>>>> fuel-master,
>>>> >>> >> there
>>>> >>> >> should be an option either to enable or disable it somewhere in
>>>> the
>>>> >>> >>fuelmenu
>>>> >>> >>
>>>> >>> >> Any thoughts?
>>>> >>>
>>>> >>> Configuration of the "local" zabbix server would need to be an
>>>> option,
>>>> >>> yes.  It's very possible that you'd have a situation where the
>>>> "local"
>>>> >>> zabbix server is watching some clouds while a dedicated zabbix
>>>> server
>>>> >>> within one of the deployed clouds is watching only the nodes within
>>>> that
>>>> >>> cloud.  I.e. there can be a mixture.  Fuel would need to keep track
>>>> of
>>>> >>> which nodes were being monitored by which server(s).
>>>> >>>
>>>> >>> >Looks like "2 - Can Zabbix be set up in HA (even in
>>>> active/passive)?"
>>>> >>> >wasn't addressed by either of the points, e.g.:
>>>> >>> >- How can we configure HA zabbix in case of:
>>>> >>> >* one of the zabbix nodes is a master node?
>>>> >>> >* some of the zabbix nodes were externally configured by the user
>>>> >>> > himself?
>>>> >>> >* some of the zabbix nodes belong to the different envs and
>>>> combined
>>>> >>> >with the another roles.
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> 2014-02-15 1:16 GMT+04:00 Andrew Woodward <xarses@xxxxxxxxx
>>>> >>> >> <mailto:xarses@xxxxxxxxx>>:
>>>> >>> >>
>>>> >>> >>     Based on the way the data-points appear to be collected,
>>>> there is
>>>> >>> >>     definitely a point where monitoring the nodes will become too
>>>> >>> >>     intense for a node sharing roles (or even a single node)
>>>> (I've
>>>> >>> >>     personally overloaded most monitoring systems in default
>>>> >>> >>     configurations with ~200 nodes.) However a single pane of
>>>> glass to
>>>> >>> >>     view all clusters is extremely important to the operations
>>>> teams.
>>>> >>> >> It
>>>> >>> >>     is therefor necessary for us to be able to specify an
>>>> existing
>>>> >>> >>     Zabbix server that may be deployed elsewhere, or even
>>>> completely
>>>> >>> >>     outside the scope of fuel.
>>>> >>> >>
>>>> >>> >>     So to get down to it. we should be able to (1) install
>>>> Zabbix as a
>>>> >>> >>     role (2) separately of the Zabbix role configure Zabbix
>>>> agents on
>>>> >>> >>     nodes inside a cluster to point to a Zabbix node. (3) if a
>>>> Zabbix
>>>> >>> >>     role is defined in the cluster assume that it should be used
>>>> to
>>>> >>> >>     configure (2) but allow it to be replaced anyway.
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>     On Fri, Feb 14, 2014 at 11:42 AM, David Easter
>>>> >>> >> <deaster@xxxxxxxxxxxx
>>>> >>> >>     <mailto:deaster@xxxxxxxxxxxx>> wrote:
>>>> >>> >>
>>>> >>> >>         Good observation!  For a situation where the tenant for
>>>> each
>>>> >>> >>         cloud were
>>>> >>> >>         the same, that would certainly make sense, yes.  Perhaps
>>>> that
>>>> >>> >>         should even
>>>> >>> >>         be the default if it doesn't impact the abilities of the
>>>> Fuel
>>>> >>> >>         Master node.
>>>> >>> >>          I.e. install the Zabbix server on the Fuel Node by
>>>> default and
>>>> >>> >>         the Zabbix
>>>> >>> >>         agents pointing back to the Fuel Master Node by default.
>>>>  If
>>>> >>> >> the
>>>> >>> >>         operator
>>>> >>> >>         decides that a given environment needs its own dedicated
>>>> Zabbix
>>>> >>> >>         server,
>>>> >>> >>         the operator could add a role on an existing node, or a
>>>> >>> >>         standalone node,
>>>> >>> >>         for a Zabbix server and all agents in the environment
>>>> would
>>>> >>> >> then
>>>> >>> >>         point to
>>>> >>> >>         that deployed server.  Then the URL for the deployed
>>>> Zabbix
>>>> >>> >>         server would
>>>> >>> >>         properly appear when the environment were opened.
>>>> >>> >>
>>>> >>> >>         In answer to your question, though; were a service
>>>> provider (or
>>>> >>> >>         large IT
>>>> >>> >>         organization) to create clouds for its customers, it may
>>>> make
>>>> >>> >>         more sense
>>>> >>> >>         to provide a monitoring service that the customer could
>>>> draw
>>>> >>> >>from
>>>> >>> >>         independently of other customers.  Or in instances where
>>>> the
>>>> >>> >>         clouds are
>>>> >>> >>         highly separated from each other in terms of data
>>>> content or
>>>> >>> >>         internal
>>>> >>> >>         departments.  By creating a Zabbix server per
>>>> environment, this
>>>> >>> >>         would
>>>> >>> >>         address issues of ensuring isolation of access and data.
>>>> >>> >>
>>>> >>> >>         Thanks,
>>>> >>> >>
>>>> >>> >>         - David J. Easter
>>>> >>> >>           Product Line Manager
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>         On 2/14/14, 11:32 AM, "Dmitry Borodaenko"
>>>> >>> >>         <dborodaenko@xxxxxxxxxxxx <mailto:
>>>> dborodaenko@xxxxxxxxxxxx>>
>>>> >>> >>wrote:
>>>> >>> >>
>>>> >>> >>         >What about deploying Zabbix on the Fuel node itself?
>>>> >>> >>         >
>>>> >>> >>         >In a scenario where a large number of OpenStack
>>>> environments
>>>> >>> >> is
>>>> >>> >>         >deployed from the same Fuel node, I'm struggling to
>>>> come up
>>>> >>> >>with a
>>>> >>> >>         >scenario where operator would prefer to have a separate
>>>> >>> >>monitoring
>>>> >>> >>         >server for each environment, instead of monitoring all
>>>> >>> >>environments
>>>> >>> >>         >from a single pane of glass.
>>>> >>> >>         >
>>>> >>> >>         >-DmitryB
>>>> >>> >>         >
>>>> >>> >>         >
>>>> >>> >>         >On Fri, Feb 14, 2014 at 9:08 AM, David Easter
>>>> >>> >>         <deaster@xxxxxxxxxxxx <mailto:deaster@xxxxxxxxxxxx>>
>>>> >>> >>         >wrote:
>>>> >>> >>         >> Correct.  Just to understand, there's growing concern
>>>> from
>>>> >>> >>         our customer
>>>> >>> >>         >> base about how many servers they need in an OpenStack
>>>> >>> >>         environment that
>>>> >>> >>         >> don't add to the compute power of the environment.  We
>>>> >>> >>         already require
>>>> >>> >>         >>the
>>>> >>> >>         >> Fuel Master Node on its own server, so _requiring_ a
>>>> Zabbix
>>>> >>> >>         node of its
>>>> >>> >>         >> own will be more difficult to justify - especially in
>>>> small
>>>> >>> >>         environments
>>>> >>> >>         >> (5-10 servers).
>>>> >>> >>         >>
>>>> >>> >>         >> While it is completely proper to point out that
>>>> individual
>>>> >>> >>         services
>>>> >>> >>         >>_can_
>>>> >>> >>         >> (and perhaps should) be installed on their own nodes
>>>> for
>>>> >>> >>         performance
>>>> >>> >>         >> reasons, the option to combine them with other roles
>>>> on the
>>>> >>> >>         same server
>>>> >>> >>         >> needs to be an option when performance is not an
>>>> issue.  In
>>>> >>> >>other
>>>> >>> >>         >>words, I
>>>> >>> >>         >> should be able to have a Zabbix server running on the
>>>> same
>>>> >>> >>         server as my
>>>> >>> >>         >> compute or controller function.  I should also be
>>>> able to
>>>> >>> >>         install it on
>>>> >>> >>         >> its own standalone server if I'm concerned about
>>>> >>> >> performance.
>>>> >>> >>         >>
>>>> >>> >>         >> Thanks,
>>>> >>> >>         >>
>>>> >>> >>         >> - David J. Easter
>>>> >>> >>         >>   Product Line Manager,  Mirantis
>>>> >>> >>         >>
>>>> >>> >>         >>
>>>> >>> >>         >> On 2/13/14, 7:49 AM, "Matthew Mosesohn"
>>>> >>> >>         <mmosesohn@xxxxxxxxxxxx <mailto:mmosesohn@xxxxxxxxxxxx>>
>>>> wrote:
>>>> >>> >>         >>
>>>> >>> >>         >>>Dmitry,
>>>> >>> >>         >>>
>>>> >>> >>         >>>I think it would be logical to say that it should be
>>>> trivial
>>>> >>> >>to
>>>> >>> >>         >>>reinstall a separate Zabbit node since it doesn't
>>>> contain
>>>> >>> >> any
>>>> >>> >>         data
>>>> >>> >>         >>>critical to the production environment. I think if
>>>> you can
>>>> >>> >>         remove and
>>>> >>> >>         >>>add a new zabbix server to a deployed environment, it
>>>> would
>>>> >>> >>         meet a lot
>>>> >>> >>         >>>of our users' needs.
>>>> >>> >>         >>>
>>>> >>> >>         >>>
>>>> >>> >>         >>>On Thu, Feb 13, 2014 at 6:54 PM, Dmitry Nikishov
>>>> >>> >>         <nikishov.da@xxxxxxxxx <mailto:nikishov.da@xxxxxxxxx>>
>>>> >>> >>         >>>wrote:
>>>> >>> >>         >>>> Matthew,
>>>> >>> >>         >>>> It should be possible to install Zabbix alongside
>>>> with
>>>> >>> >>         OpenStack
>>>> >>> >>         >>>>components
>>>> >>> >>         >>>> on a controller, compute etc. However, Zabbix isn't
>>>> an
>>>> >>> >>         OpenStack
>>>> >>> >>         >>>>component,
>>>> >>> >>         >>>> it does not directly affect cluster. So there's no
>>>> need to
>>>> >>> >>         combine it
>>>> >>> >>         >>>>with
>>>> >>> >>         >>>> other roles or making it high-available. It makes
>>>> sense to
>>>> >>> >>         install it
>>>> >>> >>         >>>>on a
>>>> >>> >>         >>>> separate node.
>>>> >>> >>         >>>>
>>>> >>> >>         >>>> Bogdan,
>>>> >>> >>         >>>> I like the idea of combining metering, monitoring
>>>> and
>>>> >>> >>         logging on one
>>>> >>> >>         >>>>node.
>>>> >>> >>         >>>> However, I think it would be better to have
>>>> separate roles
>>>> >>> >>         for all
>>>> >>> >>         >>>>three of
>>>> >>> >>         >>>> them. But these roles should not conflict with each
>>>> other.
>>>> >>> >>         This way it
>>>> >>> >>         >>>>will
>>>> >>> >>         >>>> be more customizable.
>>>> >>> >>         >>>>
>>>> >>> >>         >>>> 2014-02-13 16:07 GMT+03:00 Bogdan Dobrelya
>>>> >>> >>         <bdobrelia@xxxxxxxxxxxx <mailto:bdobrelia@xxxxxxxxxxxx
>>>> >>:
>>>> >>> >>         >>>>
>>>> >>> >>         >>>>> On 02/13/2014 02:33 PM, Matthew Mosesohn wrote:
>>>> >>> >>         >>>>> > Dmitry,
>>>> >>> >>         >>>>> >
>>>> >>> >>         >>>>> > I have two questions:
>>>> >>> >>         >>>>> > 1 - Can Zabbix Server Role be combined with any
>>>> other
>>>> >>> >>         role? (like
>>>> >>> >>         >>>>> > controller)
>>>> >>> >>         >>>>> > 2 - Can Zabbix be set up in HA (even in
>>>> >>> >> active/passive)?
>>>> >>> >>         What are
>>>> >>> >>         >>>>>the
>>>> >>> >>         >>>>> > obstacles?
>>>> >>> >>         >>>>> >
>>>> >>> >>         >>>>>
>>>> >>> >>         >>>>> And even if it cannot, I believe the good point is
>>>> to use
>>>> >>> >>         it as
>>>> >>> >>         >>>>> Monitoring & Metering & Logging node, see my
>>>> comments
>>>> >>> >>         inside the doc.
>>>> >>> >>         >>>>>
>>>> >>> >>         >>>>> > On Thu, Feb 13, 2014 at 4:28 PM, Дмитрий Никишов
>>>> >>> >>         >>>>><nikishov.da@xxxxxxxxx <mailto:
>>>> nikishov.da@xxxxxxxxx>>
>>>> >>> >>         >>>>> > wrote:
>>>> >>> >>         >>>>> >> Hello everyone.
>>>> >>> >>         >>>>> >>
>>>> >>> >>         >>>>> >> I've been working on Zabbix integration for a
>>>> while
>>>> >>> >>         now, and I'd
>>>> >>> >>         >>>>>like
>>>> >>> >>         >>>>> >> know
>>>> >>> >>         >>>>> >> what do you guys think about it.
>>>> >>> >>         >>>>> >>
>>>> >>> >>         >>>>> >> The specification:
>>>> >>> >>         >>>>> >>
>>>> >>> >>         >>>>> >>
>>>> >>> >>
>>>> >>>
>>>> >>> >>>>>>> >>>>>>>
>>>> https://docs.google.com/document/d/1g9xtBrgwgpeNV5YYxTMFu2h-yIK_PC1Z
>>>> >>> >>>>>>>JD
>>>> >>> >>         >>>>>mw
>>>> >>> >>         >>>>>84RquJk
>>>> >>> >>         >>>>> >>
>>>> >>> >>         >>>>> >> Changes to fuel-library so far:
>>>> >>> >>         >>>>>https://review.openstack.org/#/c/73254/
>>>> >>> >>         >>>>> >> Changes to fuel-main and fuel-web will be
>>>> coming soon.
>>>> >>> >>         >>>>> >>
>>>> >>> >>         >>>>> >> There is an open question related to UI -
>>>> parameter
>>>> >>> >>         validation:
>>>> >>> >>         >>>>> >> - there should be only one zabbix-server node
>>>> >>> >>         >>>>> >> - "Install Zabbix" checkbox should only be
>>>> available
>>>> >>> >> if
>>>> >>> >>         there is a
>>>> >>> >>         >>>>>node
>>>> >>> >>         >>>>> >> with
>>>> >>> >>         >>>>> >> "zabbix-server" role
>>>> >>> >>         >>>>> >>
>>>> >>> >>         >>>>> >> --
>>>> >>> >>         >>>>> >> Regards
>>>> >>> >>         >>>>> >> Dmitry Nikishov
>>>> >>> >>         >>>>> >>
>>>> >>> >>         >>>>> >> --
>>>> >>> >>         >>>>> >> Mailing list: https://launchpad.net/~fuel-dev
>>>> >>> >>         >>>>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>> >>> >>         >>>>> >> Unsubscribe : https://launchpad.net/~fuel-dev
>>>> >>> >>         >>>>> >> More help   :
>>>> https://help.launchpad.net/ListHelp
>>>> >>> >>         >>>>> >>
>>>> >>> >>         >>>>> >
>>>> >>> >>         >>>>>
>>>> >>> >>         >>>>>
>>>> >>> >>         >>>>> --
>>>> >>> >>         >>>>> Best regards,
>>>> >>> >>         >>>>> Bogdan Dobrelya,
>>>> >>> >>         >>>>> Skype #bogdando_at_yahoo.com
>>>> >>> >><http://bogdando_at_yahoo.com>
>>>> >>> >>         >>>>> Irc #bogdando
>>>> >>> >>         >>>>
>>>> >>> >>         >>>>
>>>> >>> >>         >>>>
>>>> >>> >>         >>>>
>>>> >>> >>         >>>> --
>>>> >>> >>         >>>> Regards
>>>> >>> >>         >>>> Dmitry Nikishov
>>>> >>> >>         >>>
>>>> >>> >>         >>>--
>>>> >>> >>         >>>Mailing list: https://launchpad.net/~fuel-dev
>>>> >>> >>         >>>Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>> >>> >>         >>>Unsubscribe : https://launchpad.net/~fuel-dev
>>>> >>> >>         >>>More help   : https://help.launchpad.net/ListHelp
>>>> >>> >>         >>
>>>> >>> >>         >>
>>>> >>> >>         >>
>>>> >>> >>         >> --
>>>> >>> >>         >> Mailing list: https://launchpad.net/~fuel-dev
>>>> >>> >>         >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>> >>> >>         >> Unsubscribe : https://launchpad.net/~fuel-dev
>>>> >>> >>         >> More help   : https://help.launchpad.net/ListHelp
>>>> >>> >>         >
>>>> >>> >>         >
>>>> >>> >>         >
>>>> >>> >>         >--
>>>> >>> >>         >Dmitry Borodaenko
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>         --
>>>> >>> >>         Mailing list: https://launchpad.net/~fuel-dev
>>>> >>> >>         Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>> >>> >>         Unsubscribe : https://launchpad.net/~fuel-dev
>>>> >>> >>         More help   : https://help.launchpad.net/ListHelp
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>     --
>>>> >>> >>     If google has done it, Google did it right!
>>>> >>> >>
>>>> >>> >>     --
>>>> >>> >>     Mailing list: https://launchpad.net/~fuel-dev
>>>> >>> >>     Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> >>> >>     <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>> >>> >>     Unsubscribe : https://launchpad.net/~fuel-dev
>>>> >>> >>     More help   : https://help.launchpad.net/ListHelp
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> --
>>>> >>> >> Regards
>>>> >>> >> Dmitry Nikishov
>>>> >>> >>
>>>> >>> >>
>>>> >>> >> This body part will be downloaded on demand.
>>>> >>> >>
>>>> >>> >
>>>> >>> >
>>>> >>> >--
>>>> >>> >Best regards,
>>>> >>> >Bogdan Dobrelya,
>>>> >>> >Skype #bogdando_at_yahoo.com
>>>> >>> >Irc #bogdando
>>>> >>> >
>>>> >>> >--
>>>> >>> >Mailing list: https://launchpad.net/~fuel-dev
>>>> >>> >Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> >>> >Unsubscribe : https://launchpad.net/~fuel-dev
>>>> >>> >More help   : https://help.launchpad.net/ListHelp
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> --
>>>> >>> Mailing list: https://launchpad.net/~fuel-dev
>>>> >>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> >>> Unsubscribe : https://launchpad.net/~fuel-dev
>>>> >>> More help   : https://help.launchpad.net/ListHelp
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Mailing list: https://launchpad.net/~fuel-dev
>>>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> >> Unsubscribe : https://launchpad.net/~fuel-dev
>>>> >> More help   : https://help.launchpad.net/ListHelp
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > Dmitry Borodaenko
>>>> >
>>>> > --
>>>> > Mailing list: https://launchpad.net/~fuel-dev
>>>> > Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> > Unsubscribe : https://launchpad.net/~fuel-dev
>>>> > More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>>
>>>> --
>>>> If google has done it, Google did it right!
>>>>
>>>> --
>>>> Mailing list: https://launchpad.net/~fuel-dev
>>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>
>>>
>>> --
>>> Mailing list: https://launchpad.net/~fuel-dev
>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>
>>
>> --
>> Regards
>> Dmitry Nikishov
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>>


-- 
Regards
Dmitry Nikishov