fuel-dev team mailing list archive

[Vladimir Kuklin <vkuklin@xxxxxxxxxxxx>]

Dmitry

It would be awesome if you could split the request into several ones. E. g.
one for server installation. One for each openstack service template. One
for adding templates to the hosts. Thus, it will allow us to merge it
smoothly without a headache.
07 марта 2014 г. 14:21 пользователь "Dmitry Nikishov" <nikishov.da@xxxxxxxxx>
написал:

> There's a basic version of my own reimplementation available for review:
> https://review.openstack.org/#/c/78919/
> Configuration templates were taken from PL team's module with their
> permission.
> It's able to deploy zabbix server and configure zabbix agents to monitor
> openstack services. It still requires lots of work so it should not be
> merged, but it would be great to get some feedback on it.
>
>
> 2014-02-21 14:55 GMT+04:00 Roman Zhnichkov <rzhnichkov@xxxxxxxxxxxx>:
>
>> Andrew,
>>   thanks for this catch, we'll investigate alternatives.
>>
>> --
>> Roman Zhnichkov
>>
>>
>>
>> On Thu, Feb 20, 2014 at 11:00 PM, Andrew Woodward <xarses@xxxxxxxxx>wrote:
>>
>>> I've reviewed most of the zabbix commit request and I see that it
>>> contains many uses of stored configs which wont work any longer since
>>> we don't have the puppet db / master anymore so there is likely a bit
>>> of work there. Additionally we should not merge it because it's
>>> licensed under AGPL which at this point will not be accepted. I've
>>> reached out to the upstream authors and have requested a change in
>>> license but this should not be expected. We should be reviewing
>>> additional module providers that have a more suitable upstream license
>>> and possibly less work to work in our environment.
>>>
>>> On Wed, Feb 19, 2014 at 11:34 AM, Dmitry Borodaenko
>>> <dborodaenko@xxxxxxxxxxxx> wrote:
>>> > David,
>>> >
>>> > As far as I understand the difference between #4 and #1 is that in the
>>> > case of #4 Zabbix server is not deployed by Fuel (pre-existing
>>> > monitoring server), so our job is to collect the server details from
>>> > the operator and configure Zabbix agents to report to that server.
>>> >
>>> > -DmitryB
>>> >
>>> > On Wed, Feb 19, 2014 at 6:56 AM, David Easter <deaster@xxxxxxxxxxxx>
>>> wrote:
>>> >> Use cases 1 & 2 are fine for MVP in the first release, yes.
>>> >>
>>> >> Would the mechanism for deploying use case #4 be the same as use case
>>> #1 -
>>> >> just configuring the server to monitor agents in multiple clouds?
>>> >>
>>> >> Thanks,
>>> >>
>>> >> - David J. Easter
>>> >>   Product Line Manager,  Mirantis
>>> >>
>>> >> From: Roman Zhnichkov <rzhnichkov@xxxxxxxxxxxx>
>>> >> Date: Wednesday, February 19, 2014 at 4:30 AM
>>> >> To: David Easter <deaster@xxxxxxxxxxxx>
>>> >> Cc: "fuel-dev@xxxxxxxxxxxxxxxxxxx" <fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>> >> Subject: Re: [Fuel-dev] Zabbix feature review request
>>> >>
>>> >> Guys,
>>> >>   in other words there are several use cases for Zabbix server
>>> placement:
>>> >>
>>> >> Zabbix server as separate role on dedicated hardware
>>> >> Zabbix server role combined with some other role (compute or storage,
>>> since
>>> >> Zabbix uses MySQL, so we don't want to mix OpenStack and Zabbix data
>>> in the
>>> >> same DB)
>>> >> Zabbix server on fuel-master
>>> >> Remote Zabbix server outside of OpenStack environment
>>> >> Combination of any of 4 previous items (for example, we have Zabbix
>>> server
>>> >> on fuel-master monitoring all existing environments and some other
>>> Zabbix
>>> >> servers monitoring particular OpenStack environments)
>>> >>
>>> >> Also we have general requirements:
>>> >>
>>> >> HA for Zabbix server
>>> >> Dynamic configuration (automatic node deletion when removing node
>>> from the
>>> >> environment, requires orchestration patching)
>>> >>
>>> >> As MVP I propose to have use cases 1 and 2 implemented (Zabbix server
>>> as
>>> >> separate role ans Zabbix server role combined with compute or
>>> storage) with
>>> >> dynamic configuration requirement.
>>> >>
>>> >> Use cases 3-5 as well as HA for Zabbix are subject to further
>>> >> implementation.
>>> >>
>>> >> We'll update the specification according with that.
>>> >>
>>> >> Any comments or suggestions?
>>> >>
>>> >>
>>> >> --
>>> >> Roman Zhnichkov
>>> >>
>>> >>
>>> >>
>>> >> On Tue, Feb 18, 2014 at 8:21 PM, David Easter <deaster@xxxxxxxxxxxx>
>>> wrote:
>>> >>>
>>> >>> On 2/18/14, 4:23 AM, "Bogdan Dobrelya" <bdobrelia@xxxxxxxxxxxx>
>>> wrote:
>>> >>>
>>> >>> >On 02/18/2014 12:05 PM, Dmitry Nikishov wrote:
>>> >>> >> From this conversation we got following points:
>>> >>> >> 1. It should be possible to install zabbix as a separate role
>>> >>> >> 2. Combining zabbix role with other roles should be supported as
>>> well
>>> >>> >> 3. There should be support for configuring zabbix agents to point
>>> them
>>> >>> >> to the remote zabbix server
>>> >>> >> 4. Possible zabbix installation on fuel-master?
>>> >>> >> 5. In case of one zabbix server monitoring multiple environments,
>>> nodes
>>> >>> >> should be separated into hostgroups.
>>> >>> >>
>>> >>> >> Concerns:
>>> >>> >> 1. perfomance when alot of nodes are being monitored by single
>>> server
>>> >>>
>>> >>> True.  We'd need to publish the maximum number of nodes that can be
>>> >>> monitored efficiently by one server.  If the number of monitored
>>> nodes
>>> >>> goes above that number, we'd recommend a dedicated server within the
>>> >>> largest deployed cloud (for example) to split the burden.
>>> >>>
>>> >>> >> 2. if zabbix agents were configured to connect to the remote
>>> server
>>> >>> >> (outside the environment), once this env gets deleted, it's nodes
>>> will
>>> >>> >> be rebooted into bootstrap. However, they won't be deleted from
>>> zabbix.
>>> >>> >> It's not clear how to remove them from zabbix server.
>>> >>>
>>> >>> We'd have to build in automation to do that for the user.  I.e. if a
>>> >>> remote node is deleted, update the corresponding zabbix server's
>>> info.
>>> >>>
>>> >>> >>
>>> >>> >> Open question: if there will be support for zabbix on fuel-master,
>>> >>> >> there
>>> >>> >> should be an option either to enable or disable it somewhere in
>>> the
>>> >>> >>fuelmenu
>>> >>> >>
>>> >>> >> Any thoughts?
>>> >>>
>>> >>> Configuration of the "local" zabbix server would need to be an
>>> option,
>>> >>> yes.  It's very possible that you'd have a situation where the
>>> "local"
>>> >>> zabbix server is watching some clouds while a dedicated zabbix server
>>> >>> within one of the deployed clouds is watching only the nodes within
>>> that
>>> >>> cloud.  I.e. there can be a mixture.  Fuel would need to keep track
>>> of
>>> >>> which nodes were being monitored by which server(s).
>>> >>>
>>> >>> >Looks like "2 - Can Zabbix be set up in HA (even in
>>> active/passive)?"
>>> >>> >wasn't addressed by either of the points, e.g.:
>>> >>> >- How can we configure HA zabbix in case of:
>>> >>> >* one of the zabbix nodes is a master node?
>>> >>> >* some of the zabbix nodes were externally configured by the user
>>> >>> > himself?
>>> >>> >* some of the zabbix nodes belong to the different envs and combined
>>> >>> >with the another roles.
>>> >>> >>
>>> >>> >>
>>> >>> >> 2014-02-15 1:16 GMT+04:00 Andrew Woodward <xarses@xxxxxxxxx
>>> >>> >> <mailto:xarses@xxxxxxxxx>>:
>>> >>> >>
>>> >>> >>     Based on the way the data-points appear to be collected,
>>> there is
>>> >>> >>     definitely a point where monitoring the nodes will become too
>>> >>> >>     intense for a node sharing roles (or even a single node) (I've
>>> >>> >>     personally overloaded most monitoring systems in default
>>> >>> >>     configurations with ~200 nodes.) However a single pane of
>>> glass to
>>> >>> >>     view all clusters is extremely important to the operations
>>> teams.
>>> >>> >> It
>>> >>> >>     is therefor necessary for us to be able to specify an existing
>>> >>> >>     Zabbix server that may be deployed elsewhere, or even
>>> completely
>>> >>> >>     outside the scope of fuel.
>>> >>> >>
>>> >>> >>     So to get down to it. we should be able to (1) install Zabbix
>>> as a
>>> >>> >>     role (2) separately of the Zabbix role configure Zabbix
>>> agents on
>>> >>> >>     nodes inside a cluster to point to a Zabbix node. (3) if a
>>> Zabbix
>>> >>> >>     role is defined in the cluster assume that it should be used
>>> to
>>> >>> >>     configure (2) but allow it to be replaced anyway.
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>     On Fri, Feb 14, 2014 at 11:42 AM, David Easter
>>> >>> >> <deaster@xxxxxxxxxxxx
>>> >>> >>     <mailto:deaster@xxxxxxxxxxxx>> wrote:
>>> >>> >>
>>> >>> >>         Good observation!  For a situation where the tenant for
>>> each
>>> >>> >>         cloud were
>>> >>> >>         the same, that would certainly make sense, yes.  Perhaps
>>> that
>>> >>> >>         should even
>>> >>> >>         be the default if it doesn't impact the abilities of the
>>> Fuel
>>> >>> >>         Master node.
>>> >>> >>          I.e. install the Zabbix server on the Fuel Node by
>>> default and
>>> >>> >>         the Zabbix
>>> >>> >>         agents pointing back to the Fuel Master Node by default.
>>>  If
>>> >>> >> the
>>> >>> >>         operator
>>> >>> >>         decides that a given environment needs its own dedicated
>>> Zabbix
>>> >>> >>         server,
>>> >>> >>         the operator could add a role on an existing node, or a
>>> >>> >>         standalone node,
>>> >>> >>         for a Zabbix server and all agents in the environment
>>> would
>>> >>> >> then
>>> >>> >>         point to
>>> >>> >>         that deployed server.  Then the URL for the deployed
>>> Zabbix
>>> >>> >>         server would
>>> >>> >>         properly appear when the environment were opened.
>>> >>> >>
>>> >>> >>         In answer to your question, though; were a service
>>> provider (or
>>> >>> >>         large IT
>>> >>> >>         organization) to create clouds for its customers, it may
>>> make
>>> >>> >>         more sense
>>> >>> >>         to provide a monitoring service that the customer could
>>> draw
>>> >>> >>from
>>> >>> >>         independently of other customers.  Or in instances where
>>> the
>>> >>> >>         clouds are
>>> >>> >>         highly separated from each other in terms of data content
>>> or
>>> >>> >>         internal
>>> >>> >>         departments.  By creating a Zabbix server per
>>> environment, this
>>> >>> >>         would
>>> >>> >>         address issues of ensuring isolation of access and data.
>>> >>> >>
>>> >>> >>         Thanks,
>>> >>> >>
>>> >>> >>         - David J. Easter
>>> >>> >>           Product Line Manager
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>         On 2/14/14, 11:32 AM, "Dmitry Borodaenko"
>>> >>> >>         <dborodaenko@xxxxxxxxxxxx <mailto:
>>> dborodaenko@xxxxxxxxxxxx>>
>>> >>> >>wrote:
>>> >>> >>
>>> >>> >>         >What about deploying Zabbix on the Fuel node itself?
>>> >>> >>         >
>>> >>> >>         >In a scenario where a large number of OpenStack
>>> environments
>>> >>> >> is
>>> >>> >>         >deployed from the same Fuel node, I'm struggling to come
>>> up
>>> >>> >>with a
>>> >>> >>         >scenario where operator would prefer to have a separate
>>> >>> >>monitoring
>>> >>> >>         >server for each environment, instead of monitoring all
>>> >>> >>environments
>>> >>> >>         >from a single pane of glass.
>>> >>> >>         >
>>> >>> >>         >-DmitryB
>>> >>> >>         >
>>> >>> >>         >
>>> >>> >>         >On Fri, Feb 14, 2014 at 9:08 AM, David Easter
>>> >>> >>         <deaster@xxxxxxxxxxxx <mailto:deaster@xxxxxxxxxxxx>>
>>> >>> >>         >wrote:
>>> >>> >>         >> Correct.  Just to understand, there's growing concern
>>> from
>>> >>> >>         our customer
>>> >>> >>         >> base about how many servers they need in an OpenStack
>>> >>> >>         environment that
>>> >>> >>         >> don't add to the compute power of the environment.  We
>>> >>> >>         already require
>>> >>> >>         >>the
>>> >>> >>         >> Fuel Master Node on its own server, so _requiring_ a
>>> Zabbix
>>> >>> >>         node of its
>>> >>> >>         >> own will be more difficult to justify - especially in
>>> small
>>> >>> >>         environments
>>> >>> >>         >> (5-10 servers).
>>> >>> >>         >>
>>> >>> >>         >> While it is completely proper to point out that
>>> individual
>>> >>> >>         services
>>> >>> >>         >>_can_
>>> >>> >>         >> (and perhaps should) be installed on their own nodes
>>> for
>>> >>> >>         performance
>>> >>> >>         >> reasons, the option to combine them with other roles
>>> on the
>>> >>> >>         same server
>>> >>> >>         >> needs to be an option when performance is not an
>>> issue.  In
>>> >>> >>other
>>> >>> >>         >>words, I
>>> >>> >>         >> should be able to have a Zabbix server running on the
>>> same
>>> >>> >>         server as my
>>> >>> >>         >> compute or controller function.  I should also be able
>>> to
>>> >>> >>         install it on
>>> >>> >>         >> its own standalone server if I'm concerned about
>>> >>> >> performance.
>>> >>> >>         >>
>>> >>> >>         >> Thanks,
>>> >>> >>         >>
>>> >>> >>         >> - David J. Easter
>>> >>> >>         >>   Product Line Manager,  Mirantis
>>> >>> >>         >>
>>> >>> >>         >>
>>> >>> >>         >> On 2/13/14, 7:49 AM, "Matthew Mosesohn"
>>> >>> >>         <mmosesohn@xxxxxxxxxxxx <mailto:mmosesohn@xxxxxxxxxxxx>>
>>> wrote:
>>> >>> >>         >>
>>> >>> >>         >>>Dmitry,
>>> >>> >>         >>>
>>> >>> >>         >>>I think it would be logical to say that it should be
>>> trivial
>>> >>> >>to
>>> >>> >>         >>>reinstall a separate Zabbit node since it doesn't
>>> contain
>>> >>> >> any
>>> >>> >>         data
>>> >>> >>         >>>critical to the production environment. I think if you
>>> can
>>> >>> >>         remove and
>>> >>> >>         >>>add a new zabbix server to a deployed environment, it
>>> would
>>> >>> >>         meet a lot
>>> >>> >>         >>>of our users' needs.
>>> >>> >>         >>>
>>> >>> >>         >>>
>>> >>> >>         >>>On Thu, Feb 13, 2014 at 6:54 PM, Dmitry Nikishov
>>> >>> >>         <nikishov.da@xxxxxxxxx <mailto:nikishov.da@xxxxxxxxx>>
>>> >>> >>         >>>wrote:
>>> >>> >>         >>>> Matthew,
>>> >>> >>         >>>> It should be possible to install Zabbix alongside
>>> with
>>> >>> >>         OpenStack
>>> >>> >>         >>>>components
>>> >>> >>         >>>> on a controller, compute etc. However, Zabbix isn't
>>> an
>>> >>> >>         OpenStack
>>> >>> >>         >>>>component,
>>> >>> >>         >>>> it does not directly affect cluster. So there's no
>>> need to
>>> >>> >>         combine it
>>> >>> >>         >>>>with
>>> >>> >>         >>>> other roles or making it high-available. It makes
>>> sense to
>>> >>> >>         install it
>>> >>> >>         >>>>on a
>>> >>> >>         >>>> separate node.
>>> >>> >>         >>>>
>>> >>> >>         >>>> Bogdan,
>>> >>> >>         >>>> I like the idea of combining metering, monitoring and
>>> >>> >>         logging on one
>>> >>> >>         >>>>node.
>>> >>> >>         >>>> However, I think it would be better to have separate
>>> roles
>>> >>> >>         for all
>>> >>> >>         >>>>three of
>>> >>> >>         >>>> them. But these roles should not conflict with each
>>> other.
>>> >>> >>         This way it
>>> >>> >>         >>>>will
>>> >>> >>         >>>> be more customizable.
>>> >>> >>         >>>>
>>> >>> >>         >>>> 2014-02-13 16:07 GMT+03:00 Bogdan Dobrelya
>>> >>> >>         <bdobrelia@xxxxxxxxxxxx <mailto:bdobrelia@xxxxxxxxxxxx>>:
>>> >>> >>         >>>>
>>> >>> >>         >>>>> On 02/13/2014 02:33 PM, Matthew Mosesohn wrote:
>>> >>> >>         >>>>> > Dmitry,
>>> >>> >>         >>>>> >
>>> >>> >>         >>>>> > I have two questions:
>>> >>> >>         >>>>> > 1 - Can Zabbix Server Role be combined with any
>>> other
>>> >>> >>         role? (like
>>> >>> >>         >>>>> > controller)
>>> >>> >>         >>>>> > 2 - Can Zabbix be set up in HA (even in
>>> >>> >> active/passive)?
>>> >>> >>         What are
>>> >>> >>         >>>>>the
>>> >>> >>         >>>>> > obstacles?
>>> >>> >>         >>>>> >
>>> >>> >>         >>>>>
>>> >>> >>         >>>>> And even if it cannot, I believe the good point is
>>> to use
>>> >>> >>         it as
>>> >>> >>         >>>>> Monitoring & Metering & Logging node, see my
>>> comments
>>> >>> >>         inside the doc.
>>> >>> >>         >>>>>
>>> >>> >>         >>>>> > On Thu, Feb 13, 2014 at 4:28 PM, Дмитрий Никишов
>>> >>> >>         >>>>><nikishov.da@xxxxxxxxx <mailto:nikishov.da@xxxxxxxxx
>>> >>
>>> >>> >>         >>>>> > wrote:
>>> >>> >>         >>>>> >> Hello everyone.
>>> >>> >>         >>>>> >>
>>> >>> >>         >>>>> >> I've been working on Zabbix integration for a
>>> while
>>> >>> >>         now, and I'd
>>> >>> >>         >>>>>like
>>> >>> >>         >>>>> >> know
>>> >>> >>         >>>>> >> what do you guys think about it.
>>> >>> >>         >>>>> >>
>>> >>> >>         >>>>> >> The specification:
>>> >>> >>         >>>>> >>
>>> >>> >>         >>>>> >>
>>> >>> >>
>>> >>>
>>> >>> >>>>>>> >>>>>>>
>>> https://docs.google.com/document/d/1g9xtBrgwgpeNV5YYxTMFu2h-yIK_PC1Z
>>> >>> >>>>>>>JD
>>> >>> >>         >>>>>mw
>>> >>> >>         >>>>>84RquJk
>>> >>> >>         >>>>> >>
>>> >>> >>         >>>>> >> Changes to fuel-library so far:
>>> >>> >>         >>>>>https://review.openstack.org/#/c/73254/
>>> >>> >>         >>>>> >> Changes to fuel-main and fuel-web will be coming
>>> soon.
>>> >>> >>         >>>>> >>
>>> >>> >>         >>>>> >> There is an open question related to UI -
>>> parameter
>>> >>> >>         validation:
>>> >>> >>         >>>>> >> - there should be only one zabbix-server node
>>> >>> >>         >>>>> >> - "Install Zabbix" checkbox should only be
>>> available
>>> >>> >> if
>>> >>> >>         there is a
>>> >>> >>         >>>>>node
>>> >>> >>         >>>>> >> with
>>> >>> >>         >>>>> >> "zabbix-server" role
>>> >>> >>         >>>>> >>
>>> >>> >>         >>>>> >> --
>>> >>> >>         >>>>> >> Regards
>>> >>> >>         >>>>> >> Dmitry Nikishov
>>> >>> >>         >>>>> >>
>>> >>> >>         >>>>> >> --
>>> >>> >>         >>>>> >> Mailing list: https://launchpad.net/~fuel-dev
>>> >>> >>         >>>>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>> >>> >>         >>>>> >> Unsubscribe : https://launchpad.net/~fuel-dev
>>> >>> >>         >>>>> >> More help   :
>>> https://help.launchpad.net/ListHelp
>>> >>> >>         >>>>> >>
>>> >>> >>         >>>>> >
>>> >>> >>         >>>>>
>>> >>> >>         >>>>>
>>> >>> >>         >>>>> --
>>> >>> >>         >>>>> Best regards,
>>> >>> >>         >>>>> Bogdan Dobrelya,
>>> >>> >>         >>>>> Skype #bogdando_at_yahoo.com
>>> >>> >><http://bogdando_at_yahoo.com>
>>> >>> >>         >>>>> Irc #bogdando
>>> >>> >>         >>>>
>>> >>> >>         >>>>
>>> >>> >>         >>>>
>>> >>> >>         >>>>
>>> >>> >>         >>>> --
>>> >>> >>         >>>> Regards
>>> >>> >>         >>>> Dmitry Nikishov
>>> >>> >>         >>>
>>> >>> >>         >>>--
>>> >>> >>         >>>Mailing list: https://launchpad.net/~fuel-dev
>>> >>> >>         >>>Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>> >>> >>         >>>Unsubscribe : https://launchpad.net/~fuel-dev
>>> >>> >>         >>>More help   : https://help.launchpad.net/ListHelp
>>> >>> >>         >>
>>> >>> >>         >>
>>> >>> >>         >>
>>> >>> >>         >> --
>>> >>> >>         >> Mailing list: https://launchpad.net/~fuel-dev
>>> >>> >>         >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>> >>> >>         >> Unsubscribe : https://launchpad.net/~fuel-dev
>>> >>> >>         >> More help   : https://help.launchpad.net/ListHelp
>>> >>> >>         >
>>> >>> >>         >
>>> >>> >>         >
>>> >>> >>         >--
>>> >>> >>         >Dmitry Borodaenko
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>         --
>>> >>> >>         Mailing list: https://launchpad.net/~fuel-dev
>>> >>> >>         Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>> >>> >>         Unsubscribe : https://launchpad.net/~fuel-dev
>>> >>> >>         More help   : https://help.launchpad.net/ListHelp
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>     --
>>> >>> >>     If google has done it, Google did it right!
>>> >>> >>
>>> >>> >>     --
>>> >>> >>     Mailing list: https://launchpad.net/~fuel-dev
>>> >>> >>     Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> >>> >>     <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>> >>> >>     Unsubscribe : https://launchpad.net/~fuel-dev
>>> >>> >>     More help   : https://help.launchpad.net/ListHelp
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >>
>>> >>> >> --
>>> >>> >> Regards
>>> >>> >> Dmitry Nikishov
>>> >>> >>
>>> >>> >>
>>> >>> >> This body part will be downloaded on demand.
>>> >>> >>
>>> >>> >
>>> >>> >
>>> >>> >--
>>> >>> >Best regards,
>>> >>> >Bogdan Dobrelya,
>>> >>> >Skype #bogdando_at_yahoo.com
>>> >>> >Irc #bogdando
>>> >>> >
>>> >>> >--
>>> >>> >Mailing list: https://launchpad.net/~fuel-dev
>>> >>> >Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> >>> >Unsubscribe : https://launchpad.net/~fuel-dev
>>> >>> >More help   : https://help.launchpad.net/ListHelp
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Mailing list: https://launchpad.net/~fuel-dev
>>> >>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> >>> Unsubscribe : https://launchpad.net/~fuel-dev
>>> >>> More help   : https://help.launchpad.net/ListHelp
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Mailing list: https://launchpad.net/~fuel-dev
>>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> >> Unsubscribe : https://launchpad.net/~fuel-dev
>>> >> More help   : https://help.launchpad.net/ListHelp
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > Dmitry Borodaenko
>>> >
>>> > --
>>> > Mailing list: https://launchpad.net/~fuel-dev
>>> > Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> > Unsubscribe : https://launchpad.net/~fuel-dev
>>> > More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>>>
>>> --
>>> If google has done it, Google did it right!
>>>
>>> --
>>> Mailing list: https://launchpad.net/~fuel-dev
>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>
>>
>> --
>> Mailing list: https://launchpad.net/~fuel-dev
>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~fuel-dev
>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>
>
> --
> Regards
> Dmitry Nikishov
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>
>