fuel-dev team mailing list archive

[Dmitry Nikishov <nikishov.da@xxxxxxxxx>]

I have split this reimplementation into multiple commits:

https://review.openstack.org/79566    Zabbix server installation
https://review.openstack.org/81217    Add custom types for zabbix
configuration Add basic server config
https://review.openstack.org/81723    Zabbix agent installation Basic OS
monitoring
https://review.openstack.org/81754    Add nova monitoring with zabbix
https://review.openstack.org/81765    Keystone monitoring with zabbix
Glance monitoring with zabbix
https://review.openstack.org/82036    cinder and swift monitoring with
zabbix
https://review.openstack.org/82049    memcached, mysql, horizon and rabbit
monitoring with zabbix
https://review.openstack.org/82067    misc services monitoring with zabbix
https://review.openstack.org/82433    Neutron monitoring with zabbix

Would like to get feedback on these. Please note that huge line count is
caused mainly by xml templates which are used to configure monitoring.


2014-03-11 13:29 GMT+04:00 Dmitry Nikishov <nikishov.da@xxxxxxxxx>:

> Alright, there's the first part of it: server installation.
> https://review.openstack.org/#/c/79566/
>
> Would appreciate any feedback on it.
>
>
> 2014-03-07 14:40 GMT+04:00 Vladimir Kuklin <vkuklin@xxxxxxxxxxxx>:
>
> Dmitry
>>
>> It would be awesome if you could split the request into several ones. E.
>> g. one for server installation. One for each openstack service template.
>> One for adding templates to the hosts. Thus, it will allow us to merge it
>> smoothly without a headache.
>> 07 марта 2014 г. 14:21 пользователь "Dmitry Nikishov" <
>> nikishov.da@xxxxxxxxx> написал:
>>
>>  There's a basic version of my own reimplementation available for review:
>>> https://review.openstack.org/#/c/78919/
>>> Configuration templates were taken from PL team's module with their
>>> permission.
>>> It's able to deploy zabbix server and configure zabbix agents to monitor
>>> openstack services. It still requires lots of work so it should not be
>>> merged, but it would be great to get some feedback on it.
>>>
>>>
>>> 2014-02-21 14:55 GMT+04:00 Roman Zhnichkov <rzhnichkov@xxxxxxxxxxxx>:
>>>
>>>> Andrew,
>>>>   thanks for this catch, we'll investigate alternatives.
>>>>
>>>> --
>>>> Roman Zhnichkov
>>>>
>>>>
>>>>
>>>> On Thu, Feb 20, 2014 at 11:00 PM, Andrew Woodward <xarses@xxxxxxxxx>wrote:
>>>>
>>>>> I've reviewed most of the zabbix commit request and I see that it
>>>>> contains many uses of stored configs which wont work any longer since
>>>>> we don't have the puppet db / master anymore so there is likely a bit
>>>>> of work there. Additionally we should not merge it because it's
>>>>> licensed under AGPL which at this point will not be accepted. I've
>>>>> reached out to the upstream authors and have requested a change in
>>>>> license but this should not be expected. We should be reviewing
>>>>> additional module providers that have a more suitable upstream license
>>>>> and possibly less work to work in our environment.
>>>>>
>>>>> On Wed, Feb 19, 2014 at 11:34 AM, Dmitry Borodaenko
>>>>> <dborodaenko@xxxxxxxxxxxx> wrote:
>>>>> > David,
>>>>> >
>>>>> > As far as I understand the difference between #4 and #1 is that in
>>>>> the
>>>>> > case of #4 Zabbix server is not deployed by Fuel (pre-existing
>>>>> > monitoring server), so our job is to collect the server details from
>>>>> > the operator and configure Zabbix agents to report to that server.
>>>>> >
>>>>> > -DmitryB
>>>>> >
>>>>> > On Wed, Feb 19, 2014 at 6:56 AM, David Easter <deaster@xxxxxxxxxxxx>
>>>>> wrote:
>>>>> >> Use cases 1 & 2 are fine for MVP in the first release, yes.
>>>>> >>
>>>>> >> Would the mechanism for deploying use case #4 be the same as use
>>>>> case #1 -
>>>>> >> just configuring the server to monitor agents in multiple clouds?
>>>>> >>
>>>>> >> Thanks,
>>>>> >>
>>>>> >> - David J. Easter
>>>>> >>   Product Line Manager,  Mirantis
>>>>> >>
>>>>> >> From: Roman Zhnichkov <rzhnichkov@xxxxxxxxxxxx>
>>>>> >> Date: Wednesday, February 19, 2014 at 4:30 AM
>>>>> >> To: David Easter <deaster@xxxxxxxxxxxx>
>>>>> >> Cc: "fuel-dev@xxxxxxxxxxxxxxxxxxx" <fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> >> Subject: Re: [Fuel-dev] Zabbix feature review request
>>>>> >>
>>>>> >> Guys,
>>>>> >>   in other words there are several use cases for Zabbix server
>>>>> placement:
>>>>> >>
>>>>> >> Zabbix server as separate role on dedicated hardware
>>>>> >> Zabbix server role combined with some other role (compute or
>>>>> storage, since
>>>>> >> Zabbix uses MySQL, so we don't want to mix OpenStack and Zabbix
>>>>> data in the
>>>>> >> same DB)
>>>>> >> Zabbix server on fuel-master
>>>>> >> Remote Zabbix server outside of OpenStack environment
>>>>> >> Combination of any of 4 previous items (for example, we have Zabbix
>>>>> server
>>>>> >> on fuel-master monitoring all existing environments and some other
>>>>> Zabbix
>>>>> >> servers monitoring particular OpenStack environments)
>>>>> >>
>>>>> >> Also we have general requirements:
>>>>> >>
>>>>> >> HA for Zabbix server
>>>>> >> Dynamic configuration (automatic node deletion when removing node
>>>>> from the
>>>>> >> environment, requires orchestration patching)
>>>>> >>
>>>>> >> As MVP I propose to have use cases 1 and 2 implemented (Zabbix
>>>>> server as
>>>>> >> separate role ans Zabbix server role combined with compute or
>>>>> storage) with
>>>>> >> dynamic configuration requirement.
>>>>> >>
>>>>> >> Use cases 3-5 as well as HA for Zabbix are subject to further
>>>>> >> implementation.
>>>>> >>
>>>>> >> We'll update the specification according with that.
>>>>> >>
>>>>> >> Any comments or suggestions?
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Roman Zhnichkov
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Tue, Feb 18, 2014 at 8:21 PM, David Easter <deaster@xxxxxxxxxxxx>
>>>>> wrote:
>>>>> >>>
>>>>> >>> On 2/18/14, 4:23 AM, "Bogdan Dobrelya" <bdobrelia@xxxxxxxxxxxx>
>>>>> wrote:
>>>>> >>>
>>>>> >>> >On 02/18/2014 12:05 PM, Dmitry Nikishov wrote:
>>>>> >>> >> From this conversation we got following points:
>>>>> >>> >> 1. It should be possible to install zabbix as a separate role
>>>>> >>> >> 2. Combining zabbix role with other roles should be supported
>>>>> as well
>>>>> >>> >> 3. There should be support for configuring zabbix agents to
>>>>> point them
>>>>> >>> >> to the remote zabbix server
>>>>> >>> >> 4. Possible zabbix installation on fuel-master?
>>>>> >>> >> 5. In case of one zabbix server monitoring multiple
>>>>> environments, nodes
>>>>> >>> >> should be separated into hostgroups.
>>>>> >>> >>
>>>>> >>> >> Concerns:
>>>>> >>> >> 1. perfomance when alot of nodes are being monitored by single
>>>>> server
>>>>> >>>
>>>>> >>> True.  We'd need to publish the maximum number of nodes that can be
>>>>> >>> monitored efficiently by one server.  If the number of monitored
>>>>> nodes
>>>>> >>> goes above that number, we'd recommend a dedicated server within
>>>>> the
>>>>> >>> largest deployed cloud (for example) to split the burden.
>>>>> >>>
>>>>> >>> >> 2. if zabbix agents were configured to connect to the remote
>>>>> server
>>>>> >>> >> (outside the environment), once this env gets deleted, it's
>>>>> nodes will
>>>>> >>> >> be rebooted into bootstrap. However, they won't be deleted from
>>>>> zabbix.
>>>>> >>> >> It's not clear how to remove them from zabbix server.
>>>>> >>>
>>>>> >>> We'd have to build in automation to do that for the user.  I.e. if
>>>>> a
>>>>> >>> remote node is deleted, update the corresponding zabbix server's
>>>>> info.
>>>>> >>>
>>>>> >>> >>
>>>>> >>> >> Open question: if there will be support for zabbix on
>>>>> fuel-master,
>>>>> >>> >> there
>>>>> >>> >> should be an option either to enable or disable it somewhere in
>>>>> the
>>>>> >>> >>fuelmenu
>>>>> >>> >>
>>>>> >>> >> Any thoughts?
>>>>> >>>
>>>>> >>> Configuration of the "local" zabbix server would need to be an
>>>>> option,
>>>>> >>> yes.  It's very possible that you'd have a situation where the
>>>>> "local"
>>>>> >>> zabbix server is watching some clouds while a dedicated zabbix
>>>>> server
>>>>> >>> within one of the deployed clouds is watching only the nodes
>>>>> within that
>>>>> >>> cloud.  I.e. there can be a mixture.  Fuel would need to keep
>>>>> track of
>>>>> >>> which nodes were being monitored by which server(s).
>>>>> >>>
>>>>> >>> >Looks like "2 - Can Zabbix be set up in HA (even in
>>>>> active/passive)?"
>>>>> >>> >wasn't addressed by either of the points, e.g.:
>>>>> >>> >- How can we configure HA zabbix in case of:
>>>>> >>> >* one of the zabbix nodes is a master node?
>>>>> >>> >* some of the zabbix nodes were externally configured by the user
>>>>> >>> > himself?
>>>>> >>> >* some of the zabbix nodes belong to the different envs and
>>>>> combined
>>>>> >>> >with the another roles.
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >> 2014-02-15 1:16 GMT+04:00 Andrew Woodward <xarses@xxxxxxxxx
>>>>> >>> >> <mailto:xarses@xxxxxxxxx>>:
>>>>> >>> >>
>>>>> >>> >>     Based on the way the data-points appear to be collected,
>>>>> there is
>>>>> >>> >>     definitely a point where monitoring the nodes will become
>>>>> too
>>>>> >>> >>     intense for a node sharing roles (or even a single node)
>>>>> (I've
>>>>> >>> >>     personally overloaded most monitoring systems in default
>>>>> >>> >>     configurations with ~200 nodes.) However a single pane of
>>>>> glass to
>>>>> >>> >>     view all clusters is extremely important to the operations
>>>>> teams.
>>>>> >>> >> It
>>>>> >>> >>     is therefor necessary for us to be able to specify an
>>>>> existing
>>>>> >>> >>     Zabbix server that may be deployed elsewhere, or even
>>>>> completely
>>>>> >>> >>     outside the scope of fuel.
>>>>> >>> >>
>>>>> >>> >>     So to get down to it. we should be able to (1) install
>>>>> Zabbix as a
>>>>> >>> >>     role (2) separately of the Zabbix role configure Zabbix
>>>>> agents on
>>>>> >>> >>     nodes inside a cluster to point to a Zabbix node. (3) if a
>>>>> Zabbix
>>>>> >>> >>     role is defined in the cluster assume that it should be
>>>>> used to
>>>>> >>> >>     configure (2) but allow it to be replaced anyway.
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>     On Fri, Feb 14, 2014 at 11:42 AM, David Easter
>>>>> >>> >> <deaster@xxxxxxxxxxxx
>>>>> >>> >>     <mailto:deaster@xxxxxxxxxxxx>> wrote:
>>>>> >>> >>
>>>>> >>> >>         Good observation!  For a situation where the tenant for
>>>>> each
>>>>> >>> >>         cloud were
>>>>> >>> >>         the same, that would certainly make sense, yes.
>>>>>  Perhaps that
>>>>> >>> >>         should even
>>>>> >>> >>         be the default if it doesn't impact the abilities of
>>>>> the Fuel
>>>>> >>> >>         Master node.
>>>>> >>> >>          I.e. install the Zabbix server on the Fuel Node by
>>>>> default and
>>>>> >>> >>         the Zabbix
>>>>> >>> >>         agents pointing back to the Fuel Master Node by
>>>>> default.  If
>>>>> >>> >> the
>>>>> >>> >>         operator
>>>>> >>> >>         decides that a given environment needs its own
>>>>> dedicated Zabbix
>>>>> >>> >>         server,
>>>>> >>> >>         the operator could add a role on an existing node, or a
>>>>> >>> >>         standalone node,
>>>>> >>> >>         for a Zabbix server and all agents in the environment
>>>>> would
>>>>> >>> >> then
>>>>> >>> >>         point to
>>>>> >>> >>         that deployed server.  Then the URL for the deployed
>>>>> Zabbix
>>>>> >>> >>         server would
>>>>> >>> >>         properly appear when the environment were opened.
>>>>> >>> >>
>>>>> >>> >>         In answer to your question, though; were a service
>>>>> provider (or
>>>>> >>> >>         large IT
>>>>> >>> >>         organization) to create clouds for its customers, it
>>>>> may make
>>>>> >>> >>         more sense
>>>>> >>> >>         to provide a monitoring service that the customer could
>>>>> draw
>>>>> >>> >>from
>>>>> >>> >>         independently of other customers.  Or in instances
>>>>> where the
>>>>> >>> >>         clouds are
>>>>> >>> >>         highly separated from each other in terms of data
>>>>> content or
>>>>> >>> >>         internal
>>>>> >>> >>         departments.  By creating a Zabbix server per
>>>>> environment, this
>>>>> >>> >>         would
>>>>> >>> >>         address issues of ensuring isolation of access and data.
>>>>> >>> >>
>>>>> >>> >>         Thanks,
>>>>> >>> >>
>>>>> >>> >>         - David J. Easter
>>>>> >>> >>           Product Line Manager
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>         On 2/14/14, 11:32 AM, "Dmitry Borodaenko"
>>>>> >>> >>         <dborodaenko@xxxxxxxxxxxx <mailto:
>>>>> dborodaenko@xxxxxxxxxxxx>>
>>>>> >>> >>wrote:
>>>>> >>> >>
>>>>> >>> >>         >What about deploying Zabbix on the Fuel node itself?
>>>>> >>> >>         >
>>>>> >>> >>         >In a scenario where a large number of OpenStack
>>>>> environments
>>>>> >>> >> is
>>>>> >>> >>         >deployed from the same Fuel node, I'm struggling to
>>>>> come up
>>>>> >>> >>with a
>>>>> >>> >>         >scenario where operator would prefer to have a separate
>>>>> >>> >>monitoring
>>>>> >>> >>         >server for each environment, instead of monitoring all
>>>>> >>> >>environments
>>>>> >>> >>         >from a single pane of glass.
>>>>> >>> >>         >
>>>>> >>> >>         >-DmitryB
>>>>> >>> >>         >
>>>>> >>> >>         >
>>>>> >>> >>         >On Fri, Feb 14, 2014 at 9:08 AM, David Easter
>>>>> >>> >>         <deaster@xxxxxxxxxxxx <mailto:deaster@xxxxxxxxxxxx>>
>>>>> >>> >>         >wrote:
>>>>> >>> >>         >> Correct.  Just to understand, there's growing
>>>>> concern from
>>>>> >>> >>         our customer
>>>>> >>> >>         >> base about how many servers they need in an OpenStack
>>>>> >>> >>         environment that
>>>>> >>> >>         >> don't add to the compute power of the environment.
>>>>>  We
>>>>> >>> >>         already require
>>>>> >>> >>         >>the
>>>>> >>> >>         >> Fuel Master Node on its own server, so _requiring_ a
>>>>> Zabbix
>>>>> >>> >>         node of its
>>>>> >>> >>         >> own will be more difficult to justify - especially
>>>>> in small
>>>>> >>> >>         environments
>>>>> >>> >>         >> (5-10 servers).
>>>>> >>> >>         >>
>>>>> >>> >>         >> While it is completely proper to point out that
>>>>> individual
>>>>> >>> >>         services
>>>>> >>> >>         >>_can_
>>>>> >>> >>         >> (and perhaps should) be installed on their own nodes
>>>>> for
>>>>> >>> >>         performance
>>>>> >>> >>         >> reasons, the option to combine them with other roles
>>>>> on the
>>>>> >>> >>         same server
>>>>> >>> >>         >> needs to be an option when performance is not an
>>>>> issue.  In
>>>>> >>> >>other
>>>>> >>> >>         >>words, I
>>>>> >>> >>         >> should be able to have a Zabbix server running on
>>>>> the same
>>>>> >>> >>         server as my
>>>>> >>> >>         >> compute or controller function.  I should also be
>>>>> able to
>>>>> >>> >>         install it on
>>>>> >>> >>         >> its own standalone server if I'm concerned about
>>>>> >>> >> performance.
>>>>> >>> >>         >>
>>>>> >>> >>         >> Thanks,
>>>>> >>> >>         >>
>>>>> >>> >>         >> - David J. Easter
>>>>> >>> >>         >>   Product Line Manager,  Mirantis
>>>>> >>> >>         >>
>>>>> >>> >>         >>
>>>>> >>> >>         >> On 2/13/14, 7:49 AM, "Matthew Mosesohn"
>>>>> >>> >>         <mmosesohn@xxxxxxxxxxxx <mailto:mmosesohn@xxxxxxxxxxxx>>
>>>>> wrote:
>>>>> >>> >>         >>
>>>>> >>> >>         >>>Dmitry,
>>>>> >>> >>         >>>
>>>>> >>> >>         >>>I think it would be logical to say that it should be
>>>>> trivial
>>>>> >>> >>to
>>>>> >>> >>         >>>reinstall a separate Zabbit node since it doesn't
>>>>> contain
>>>>> >>> >> any
>>>>> >>> >>         data
>>>>> >>> >>         >>>critical to the production environment. I think if
>>>>> you can
>>>>> >>> >>         remove and
>>>>> >>> >>         >>>add a new zabbix server to a deployed environment,
>>>>> it would
>>>>> >>> >>         meet a lot
>>>>> >>> >>         >>>of our users' needs.
>>>>> >>> >>         >>>
>>>>> >>> >>         >>>
>>>>> >>> >>         >>>On Thu, Feb 13, 2014 at 6:54 PM, Dmitry Nikishov
>>>>> >>> >>         <nikishov.da@xxxxxxxxx <mailto:nikishov.da@xxxxxxxxx>>
>>>>> >>> >>         >>>wrote:
>>>>> >>> >>         >>>> Matthew,
>>>>> >>> >>         >>>> It should be possible to install Zabbix alongside
>>>>> with
>>>>> >>> >>         OpenStack
>>>>> >>> >>         >>>>components
>>>>> >>> >>         >>>> on a controller, compute etc. However, Zabbix
>>>>> isn't an
>>>>> >>> >>         OpenStack
>>>>> >>> >>         >>>>component,
>>>>> >>> >>         >>>> it does not directly affect cluster. So there's no
>>>>> need to
>>>>> >>> >>         combine it
>>>>> >>> >>         >>>>with
>>>>> >>> >>         >>>> other roles or making it high-available. It makes
>>>>> sense to
>>>>> >>> >>         install it
>>>>> >>> >>         >>>>on a
>>>>> >>> >>         >>>> separate node.
>>>>> >>> >>         >>>>
>>>>> >>> >>         >>>> Bogdan,
>>>>> >>> >>         >>>> I like the idea of combining metering, monitoring
>>>>> and
>>>>> >>> >>         logging on one
>>>>> >>> >>         >>>>node.
>>>>> >>> >>         >>>> However, I think it would be better to have
>>>>> separate roles
>>>>> >>> >>         for all
>>>>> >>> >>         >>>>three of
>>>>> >>> >>         >>>> them. But these roles should not conflict with
>>>>> each other.
>>>>> >>> >>         This way it
>>>>> >>> >>         >>>>will
>>>>> >>> >>         >>>> be more customizable.
>>>>> >>> >>         >>>>
>>>>> >>> >>         >>>> 2014-02-13 16:07 GMT+03:00 Bogdan Dobrelya
>>>>> >>> >>         <bdobrelia@xxxxxxxxxxxx <mailto:bdobrelia@xxxxxxxxxxxx
>>>>> >>:
>>>>> >>> >>         >>>>
>>>>> >>> >>         >>>>> On 02/13/2014 02:33 PM, Matthew Mosesohn wrote:
>>>>> >>> >>         >>>>> > Dmitry,
>>>>> >>> >>         >>>>> >
>>>>> >>> >>         >>>>> > I have two questions:
>>>>> >>> >>         >>>>> > 1 - Can Zabbix Server Role be combined with any
>>>>> other
>>>>> >>> >>         role? (like
>>>>> >>> >>         >>>>> > controller)
>>>>> >>> >>         >>>>> > 2 - Can Zabbix be set up in HA (even in
>>>>> >>> >> active/passive)?
>>>>> >>> >>         What are
>>>>> >>> >>         >>>>>the
>>>>> >>> >>         >>>>> > obstacles?
>>>>> >>> >>         >>>>> >
>>>>> >>> >>         >>>>>
>>>>> >>> >>         >>>>> And even if it cannot, I believe the good point
>>>>> is to use
>>>>> >>> >>         it as
>>>>> >>> >>         >>>>> Monitoring & Metering & Logging node, see my
>>>>> comments
>>>>> >>> >>         inside the doc.
>>>>> >>> >>         >>>>>
>>>>> >>> >>         >>>>> > On Thu, Feb 13, 2014 at 4:28 PM, Дмитрий Никишов
>>>>> >>> >>         >>>>><nikishov.da@xxxxxxxxx <mailto:
>>>>> nikishov.da@xxxxxxxxx>>
>>>>> >>> >>         >>>>> > wrote:
>>>>> >>> >>         >>>>> >> Hello everyone.
>>>>> >>> >>         >>>>> >>
>>>>> >>> >>         >>>>> >> I've been working on Zabbix integration for a
>>>>> while
>>>>> >>> >>         now, and I'd
>>>>> >>> >>         >>>>>like
>>>>> >>> >>         >>>>> >> know
>>>>> >>> >>         >>>>> >> what do you guys think about it.
>>>>> >>> >>         >>>>> >>
>>>>> >>> >>         >>>>> >> The specification:
>>>>> >>> >>         >>>>> >>
>>>>> >>> >>         >>>>> >>
>>>>> >>> >>
>>>>> >>>
>>>>> >>> >>>>>>> >>>>>>>
>>>>> https://docs.google.com/document/d/1g9xtBrgwgpeNV5YYxTMFu2h-yIK_PC1Z
>>>>> >>> >>>>>>>JD
>>>>> >>> >>         >>>>>mw
>>>>> >>> >>         >>>>>84RquJk
>>>>> >>> >>         >>>>> >>
>>>>> >>> >>         >>>>> >> Changes to fuel-library so far:
>>>>> >>> >>         >>>>>https://review.openstack.org/#/c/73254/
>>>>> >>> >>         >>>>> >> Changes to fuel-main and fuel-web will be
>>>>> coming soon.
>>>>> >>> >>         >>>>> >>
>>>>> >>> >>         >>>>> >> There is an open question related to UI -
>>>>> parameter
>>>>> >>> >>         validation:
>>>>> >>> >>         >>>>> >> - there should be only one zabbix-server node
>>>>> >>> >>         >>>>> >> - "Install Zabbix" checkbox should only be
>>>>> available
>>>>> >>> >> if
>>>>> >>> >>         there is a
>>>>> >>> >>         >>>>>node
>>>>> >>> >>         >>>>> >> with
>>>>> >>> >>         >>>>> >> "zabbix-server" role
>>>>> >>> >>         >>>>> >>
>>>>> >>> >>         >>>>> >> --
>>>>> >>> >>         >>>>> >> Regards
>>>>> >>> >>         >>>>> >> Dmitry Nikishov
>>>>> >>> >>         >>>>> >>
>>>>> >>> >>         >>>>> >> --
>>>>> >>> >>         >>>>> >> Mailing list: https://launchpad.net/~fuel-dev
>>>>> >>> >>         >>>>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> >>> >>         >>>>> >> Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >>> >>         >>>>> >> More help   :
>>>>> https://help.launchpad.net/ListHelp
>>>>> >>> >>         >>>>> >>
>>>>> >>> >>         >>>>> >
>>>>> >>> >>         >>>>>
>>>>> >>> >>         >>>>>
>>>>> >>> >>         >>>>> --
>>>>> >>> >>         >>>>> Best regards,
>>>>> >>> >>         >>>>> Bogdan Dobrelya,
>>>>> >>> >>         >>>>> Skype #bogdando_at_yahoo.com
>>>>> >>> >><http://bogdando_at_yahoo.com>
>>>>> >>> >>         >>>>> Irc #bogdando
>>>>> >>> >>         >>>>
>>>>> >>> >>         >>>>
>>>>> >>> >>         >>>>
>>>>> >>> >>         >>>>
>>>>> >>> >>         >>>> --
>>>>> >>> >>         >>>> Regards
>>>>> >>> >>         >>>> Dmitry Nikishov
>>>>> >>> >>         >>>
>>>>> >>> >>         >>>--
>>>>> >>> >>         >>>Mailing list: https://launchpad.net/~fuel-dev
>>>>> >>> >>         >>>Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> >>> >>         >>>Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >>> >>         >>>More help   : https://help.launchpad.net/ListHelp
>>>>> >>> >>         >>
>>>>> >>> >>         >>
>>>>> >>> >>         >>
>>>>> >>> >>         >> --
>>>>> >>> >>         >> Mailing list: https://launchpad.net/~fuel-dev
>>>>> >>> >>         >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> >>> >>         >> Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >>> >>         >> More help   : https://help.launchpad.net/ListHelp
>>>>> >>> >>         >
>>>>> >>> >>         >
>>>>> >>> >>         >
>>>>> >>> >>         >--
>>>>> >>> >>         >Dmitry Borodaenko
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>         --
>>>>> >>> >>         Mailing list: https://launchpad.net/~fuel-dev
>>>>> >>> >>         Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >>> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> >>> >>         Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >>> >>         More help   : https://help.launchpad.net/ListHelp
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>     --
>>>>> >>> >>     If google has done it, Google did it right!
>>>>> >>> >>
>>>>> >>> >>     --
>>>>> >>> >>     Mailing list: https://launchpad.net/~fuel-dev
>>>>> >>> >>     Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >>> >>     <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
>>>>> >>> >>     Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >>> >>     More help   : https://help.launchpad.net/ListHelp
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >> --
>>>>> >>> >> Regards
>>>>> >>> >> Dmitry Nikishov
>>>>> >>> >>
>>>>> >>> >>
>>>>> >>> >> This body part will be downloaded on demand.
>>>>> >>> >>
>>>>> >>> >
>>>>> >>> >
>>>>> >>> >--
>>>>> >>> >Best regards,
>>>>> >>> >Bogdan Dobrelya,
>>>>> >>> >Skype #bogdando_at_yahoo.com
>>>>> >>> >Irc #bogdando
>>>>> >>> >
>>>>> >>> >--
>>>>> >>> >Mailing list: https://launchpad.net/~fuel-dev
>>>>> >>> >Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >>> >Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >>> >More help   : https://help.launchpad.net/ListHelp
>>>>> >>>
>>>>> >>>
>>>>> >>>
>>>>> >>> --
>>>>> >>> Mailing list: https://launchpad.net/~fuel-dev
>>>>> >>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >>> Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >>> More help   : https://help.launchpad.net/ListHelp
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> --
>>>>> >> Mailing list: https://launchpad.net/~fuel-dev
>>>>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> >> Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> >> More help   : https://help.launchpad.net/ListHelp
>>>>> >>
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Dmitry Borodaenko
>>>>> >
>>>>> > --
>>>>> > Mailing list: https://launchpad.net/~fuel-dev
>>>>> > Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> > Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> > More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> If google has done it, Google did it right!
>>>>>
>>>>> --
>>>>> Mailing list: https://launchpad.net/~fuel-dev
>>>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>
>>>>
>>>>
>>>> --
>>>> Mailing list: https://launchpad.net/~fuel-dev
>>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards
>>> Dmitry Nikishov
>>>
>>> --
>>> Mailing list: https://launchpad.net/~fuel-dev
>>> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~fuel-dev
>>> More help   : https://help.launchpad.net/ListHelp
>>>
>>>
>
>
> --
> Regards
> Dmitry Nikishov
>



-- 
Regards
Dmitry Nikishov