fuel-dev team mailing list archive

[Roman Zhnichkov <rzhnichkov@xxxxxxxxxxxx>]

Guys,
  in other words there are several use cases for Zabbix server placement:

   1. Zabbix server as separate role on dedicated hardware
   2. Zabbix server role combined with some other role (compute or storage,
   since Zabbix uses MySQL, so we don't want to mix OpenStack and Zabbix data
   in the same DB)
   3. Zabbix server on fuel-master
   4. Remote Zabbix server outside of OpenStack environment
   5. Combination of any of 4 previous items (for example, we have Zabbix
   server on fuel-master monitoring all existing environments and some other
   Zabbix servers monitoring particular OpenStack environments)

Also we have general requirements:

   - HA for Zabbix server
   - Dynamic configuration (automatic node deletion when removing node from
   the environment, requires orchestration patching)

As MVP I propose to have use cases 1 and 2 implemented (Zabbix server as
separate role ans Zabbix server role combined with compute or storage) with
dynamic configuration requirement.

Use cases 3-5 as well as HA for Zabbix are subject to further
implementation.

We'll update the specification according with that.

Any comments or suggestions?


--
Roman Zhnichkov



On Tue, Feb 18, 2014 at 8:21 PM, David Easter <deaster@xxxxxxxxxxxx> wrote:

> On 2/18/14, 4:23 AM, "Bogdan Dobrelya" <bdobrelia@xxxxxxxxxxxx> wrote:
>
> >On 02/18/2014 12:05 PM, Dmitry Nikishov wrote:
> >> From this conversation we got following points:
> >> 1. It should be possible to install zabbix as a separate role
> >> 2. Combining zabbix role with other roles should be supported as well
> >> 3. There should be support for configuring zabbix agents to point them
> >> to the remote zabbix server
> >> 4. Possible zabbix installation on fuel-master?
> >> 5. In case of one zabbix server monitoring multiple environments, nodes
> >> should be separated into hostgroups.
> >>
> >> Concerns:
> >> 1. perfomance when alot of nodes are being monitored by single server
>
> True.  We’d need to publish the maximum number of nodes that can be
> monitored efficiently by one server.  If the number of monitored nodes
> goes above that number, we’d recommend a dedicated server within the
> largest deployed cloud (for example) to split the burden.
>
> >> 2. if zabbix agents were configured to connect to the remote server
> >> (outside the environment), once this env gets deleted, it's nodes will
> >> be rebooted into bootstrap. However, they won't be deleted from zabbix.
> >> It's not clear how to remove them from zabbix server.
>
> We’d have to build in automation to do that for the user.  I.e. if a
> remote node is deleted, update the corresponding zabbix server’s info.
>
> >>
> >> Open question: if there will be support for zabbix on fuel-master, there
> >> should be an option either to enable or disable it somewhere in the
> >>fuelmenu
> >>
> >> Any thoughts?
>
> Configuration of the “local” zabbix server would need to be an option,
> yes.  It’s very possible that you’d have a situation where the “local”
> zabbix server is watching some clouds while a dedicated zabbix server
> within one of the deployed clouds is watching only the nodes within that
> cloud.  I.e. there can be a mixture.  Fuel would need to keep track of
> which nodes were being monitored by which server(s).
>
> >Looks like "2 - Can Zabbix be set up in HA (even in active/passive)?"
> >wasn't addressed by either of the points, e.g.:
> >- How can we configure HA zabbix in case of:
> >* one of the zabbix nodes is a master node?
> >* some of the zabbix nodes were externally configured by the user himself?
> >* some of the zabbix nodes belong to the different envs and combined
> >with the another roles.
> >>
> >>
> >> 2014-02-15 1:16 GMT+04:00 Andrew Woodward <xarses@xxxxxxxxx
> >> <mailto:xarses@xxxxxxxxx>>:
> >>
> >>     Based on the way the data-points appear to be collected, there is
> >>     definitely a point where monitoring the nodes will become too
> >>     intense for a node sharing roles (or even a single node) (I've
> >>     personally overloaded most monitoring systems in default
> >>     configurations with ~200 nodes.) However a single pane of glass to
> >>     view all clusters is extremely important to the operations teams. It
> >>     is therefor necessary for us to be able to specify an existing
> >>     Zabbix server that may be deployed elsewhere, or even completely
> >>     outside the scope of fuel.
> >>
> >>     So to get down to it. we should be able to (1) install Zabbix as a
> >>     role (2) separately of the Zabbix role configure Zabbix agents on
> >>     nodes inside a cluster to point to a Zabbix node. (3) if a Zabbix
> >>     role is defined in the cluster assume that it should be used to
> >>     configure (2) but allow it to be replaced anyway.
> >>
> >>
> >>
> >>     On Fri, Feb 14, 2014 at 11:42 AM, David Easter <
> deaster@xxxxxxxxxxxx
> >>     <mailto:deaster@xxxxxxxxxxxx>> wrote:
> >>
> >>         Good observation!  For a situation where the tenant for each
> >>         cloud were
> >>         the same, that would certainly make sense, yes.  Perhaps that
> >>         should even
> >>         be the default if it doesn’t impact the abilities of the Fuel
> >>         Master node.
> >>          I.e. install the Zabbix server on the Fuel Node by default and
> >>         the Zabbix
> >>         agents pointing back to the Fuel Master Node by default.  If the
> >>         operator
> >>         decides that a given environment needs its own dedicated Zabbix
> >>         server,
> >>         the operator could add a role on an existing node, or a
> >>         standalone node,
> >>         for a Zabbix server and all agents in the environment would then
> >>         point to
> >>         that deployed server.  Then the URL for the deployed Zabbix
> >>         server would
> >>         properly appear when the environment were opened.
> >>
> >>         In answer to your question, though; were a service provider (or
> >>         large IT
> >>         organization) to create clouds for its customers, it may make
> >>         more sense
> >>         to provide a monitoring service that the customer could draw
> >>from
> >>         independently of other customers.  Or in instances where the
> >>         clouds are
> >>         highly separated from each other in terms of data content or
> >>         internal
> >>         departments.  By creating a Zabbix server per environment, this
> >>         would
> >>         address issues of ensuring isolation of access and data.
> >>
> >>         Thanks,
> >>
> >>         - David J. Easter
> >>           Product Line Manager
> >>
> >>
> >>
> >>         On 2/14/14, 11:32 AM, "Dmitry Borodaenko"
> >>         <dborodaenko@xxxxxxxxxxxx <mailto:dborodaenko@xxxxxxxxxxxx>>
> >>wrote:
> >>
> >>         >What about deploying Zabbix on the Fuel node itself?
> >>         >
> >>         >In a scenario where a large number of OpenStack environments is
> >>         >deployed from the same Fuel node, I'm struggling to come up
> >>with a
> >>         >scenario where operator would prefer to have a separate
> >>monitoring
> >>         >server for each environment, instead of monitoring all
> >>environments
> >>         >from a single pane of glass.
> >>         >
> >>         >-DmitryB
> >>         >
> >>         >
> >>         >On Fri, Feb 14, 2014 at 9:08 AM, David Easter
> >>         <deaster@xxxxxxxxxxxx <mailto:deaster@xxxxxxxxxxxx>>
> >>         >wrote:
> >>         >> Correct.  Just to understand, there’s growing concern from
> >>         our customer
> >>         >> base about how many servers they need in an OpenStack
> >>         environment that
> >>         >> don’t add to the compute power of the environment.  We
> >>         already require
> >>         >>the
> >>         >> Fuel Master Node on its own server, so _requiring_ a Zabbix
> >>         node of its
> >>         >> own will be more difficult to justify - especially in small
> >>         environments
> >>         >> (5-10 servers).
> >>         >>
> >>         >> While it is completely proper to point out that individual
> >>         services
> >>         >>_can_
> >>         >> (and perhaps should) be installed on their own nodes for
> >>         performance
> >>         >> reasons, the option to combine them with other roles on the
> >>         same server
> >>         >> needs to be an option when performance is not an issue.  In
> >>other
> >>         >>words, I
> >>         >> should be able to have a Zabbix server running on the same
> >>         server as my
> >>         >> compute or controller function.  I should also be able to
> >>         install it on
> >>         >> its own standalone server if I’m concerned about performance.
> >>         >>
> >>         >> Thanks,
> >>         >>
> >>         >> - David J. Easter
> >>         >>   Product Line Manager,  Mirantis
> >>         >>
> >>         >>
> >>         >> On 2/13/14, 7:49 AM, "Matthew Mosesohn"
> >>         <mmosesohn@xxxxxxxxxxxx <mailto:mmosesohn@xxxxxxxxxxxx>> wrote:
> >>         >>
> >>         >>>Dmitry,
> >>         >>>
> >>         >>>I think it would be logical to say that it should be trivial
> >>to
> >>         >>>reinstall a separate Zabbit node since it doesn't contain any
> >>         data
> >>         >>>critical to the production environment. I think if you can
> >>         remove and
> >>         >>>add a new zabbix server to a deployed environment, it would
> >>         meet a lot
> >>         >>>of our users' needs.
> >>         >>>
> >>         >>>
> >>         >>>On Thu, Feb 13, 2014 at 6:54 PM, Dmitry Nikishov
> >>         <nikishov.da@xxxxxxxxx <mailto:nikishov.da@xxxxxxxxx>>
> >>         >>>wrote:
> >>         >>>> Matthew,
> >>         >>>> It should be possible to install Zabbix alongside with
> >>         OpenStack
> >>         >>>>components
> >>         >>>> on a controller, compute etc. However, Zabbix isn't an
> >>         OpenStack
> >>         >>>>component,
> >>         >>>> it does not directly affect cluster. So there's no need to
> >>         combine it
> >>         >>>>with
> >>         >>>> other roles or making it high-available. It makes sense to
> >>         install it
> >>         >>>>on a
> >>         >>>> separate node.
> >>         >>>>
> >>         >>>> Bogdan,
> >>         >>>> I like the idea of combining metering, monitoring and
> >>         logging on one
> >>         >>>>node.
> >>         >>>> However, I think it would be better to have separate roles
> >>         for all
> >>         >>>>three of
> >>         >>>> them. But these roles should not conflict with each other.
> >>         This way it
> >>         >>>>will
> >>         >>>> be more customizable.
> >>         >>>>
> >>         >>>> 2014-02-13 16:07 GMT+03:00 Bogdan Dobrelya
> >>         <bdobrelia@xxxxxxxxxxxx <mailto:bdobrelia@xxxxxxxxxxxx>>:
> >>         >>>>
> >>         >>>>> On 02/13/2014 02:33 PM, Matthew Mosesohn wrote:
> >>         >>>>> > Dmitry,
> >>         >>>>> >
> >>         >>>>> > I have two questions:
> >>         >>>>> > 1 - Can Zabbix Server Role be combined with any other
> >>         role? (like
> >>         >>>>> > controller)
> >>         >>>>> > 2 - Can Zabbix be set up in HA (even in active/passive)?
> >>         What are
> >>         >>>>>the
> >>         >>>>> > obstacles?
> >>         >>>>> >
> >>         >>>>>
> >>         >>>>> And even if it cannot, I believe the good point is to use
> >>         it as
> >>         >>>>> Monitoring & Metering & Logging node, see my comments
> >>         inside the doc.
> >>         >>>>>
> >>         >>>>> > On Thu, Feb 13, 2014 at 4:28 PM, Дмитрий Никишов
> >>         >>>>><nikishov.da@xxxxxxxxx <mailto:nikishov.da@xxxxxxxxx>>
> >>         >>>>> > wrote:
> >>         >>>>> >> Hello everyone.
> >>         >>>>> >>
> >>         >>>>> >> I've been working on Zabbix integration for a while
> >>         now, and I'd
> >>         >>>>>like
> >>         >>>>> >> know
> >>         >>>>> >> what do you guys think about it.
> >>         >>>>> >>
> >>         >>>>> >> The specification:
> >>         >>>>> >>
> >>         >>>>> >>
> >>
> >>>>>>>
> https://docs.google.com/document/d/1g9xtBrgwgpeNV5YYxTMFu2h-yIK_PC1Z
> >>>>>>>JD
> >>         >>>>>mw
> >>         >>>>>84RquJk
> >>         >>>>> >>
> >>         >>>>> >> Changes to fuel-library so far:
> >>         >>>>>https://review.openstack.org/#/c/73254/
> >>         >>>>> >> Changes to fuel-main and fuel-web will be coming soon.
> >>         >>>>> >>
> >>         >>>>> >> There is an open question related to UI - parameter
> >>         validation:
> >>         >>>>> >> - there should be only one zabbix-server node
> >>         >>>>> >> - "Install Zabbix" checkbox should only be available if
> >>         there is a
> >>         >>>>>node
> >>         >>>>> >> with
> >>         >>>>> >> "zabbix-server" role
> >>         >>>>> >>
> >>         >>>>> >> --
> >>         >>>>> >> Regards
> >>         >>>>> >> Dmitry Nikishov
> >>         >>>>> >>
> >>         >>>>> >> --
> >>         >>>>> >> Mailing list: https://launchpad.net/~fuel-dev
> >>         >>>>> >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
> >>         >>>>> >> Unsubscribe : https://launchpad.net/~fuel-dev
> >>         >>>>> >> More help   : https://help.launchpad.net/ListHelp
> >>         >>>>> >>
> >>         >>>>> >
> >>         >>>>>
> >>         >>>>>
> >>         >>>>> --
> >>         >>>>> Best regards,
> >>         >>>>> Bogdan Dobrelya,
> >>         >>>>> Skype #bogdando_at_yahoo.com
> >><http://bogdando_at_yahoo.com>
> >>         >>>>> Irc #bogdando
> >>         >>>>
> >>         >>>>
> >>         >>>>
> >>         >>>>
> >>         >>>> --
> >>         >>>> Regards
> >>         >>>> Dmitry Nikishov
> >>         >>>
> >>         >>>--
> >>         >>>Mailing list: https://launchpad.net/~fuel-dev
> >>         >>>Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
> >>         >>>Unsubscribe : https://launchpad.net/~fuel-dev
> >>         >>>More help   : https://help.launchpad.net/ListHelp
> >>         >>
> >>         >>
> >>         >>
> >>         >> --
> >>         >> Mailing list: https://launchpad.net/~fuel-dev
> >>         >> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
> >>         >> Unsubscribe : https://launchpad.net/~fuel-dev
> >>         >> More help   : https://help.launchpad.net/ListHelp
> >>         >
> >>         >
> >>         >
> >>         >--
> >>         >Dmitry Borodaenko
> >>
> >>
> >>
> >>         --
> >>         Mailing list: https://launchpad.net/~fuel-dev
> >>         Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> >>         <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
> >>         Unsubscribe : https://launchpad.net/~fuel-dev
> >>         More help   : https://help.launchpad.net/ListHelp
> >>
> >>
> >>
> >>
> >>     --
> >>     If google has done it, Google did it right!
> >>
> >>     --
> >>     Mailing list: https://launchpad.net/~fuel-dev
> >>     Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> >>     <mailto:fuel-dev@xxxxxxxxxxxxxxxxxxx>
> >>     Unsubscribe : https://launchpad.net/~fuel-dev
> >>     More help   : https://help.launchpad.net/ListHelp
> >>
> >>
> >>
> >>
> >> --
> >> Regards
> >> Dmitry Nikishov
> >>
> >>
> >> This body part will be downloaded on demand.
> >>
> >
> >
> >--
> >Best regards,
> >Bogdan Dobrelya,
> >Skype #bogdando_at_yahoo.com
> >Irc #bogdando
> >
> >--
> >Mailing list: https://launchpad.net/~fuel-dev
> >Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> >Unsubscribe : https://launchpad.net/~fuel-dev
> >More help   : https://help.launchpad.net/ListHelp
>
>
>
> --
> Mailing list: https://launchpad.net/~fuel-dev
> Post to     : fuel-dev@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~fuel-dev
> More help   : https://help.launchpad.net/ListHelp
>