cf-charmers team mailing list archive

[Kapil Thangavelu <kapil.thangavelu@xxxxxxxxxxxxx>]

On Fri, Sep 5, 2014 at 3:06 PM, Cory Johns <cory.johns@xxxxxxxxxxxxx> wrote:

> Alexander,
>
> We should not need a new interface on the existing charms.  We will
> need a service to perform the log draining (as I noted in my last
> point), which should essentially adapt the existing
> "loggregator_trafficcontroller" interface to the "syslog" interface.
> I think this would be best to be its own charm, e.g. cf-logdrain, and
> then should be able to connect CF to anything that supports the
> "syslog" interface.  My other proposal was to add support for this
> interface to the logstash-indexer charm.
>
> The reason cf-logdrain should be its own charm and that we should
> avoid making changes to the loggregator_trafficcontrol charm is
> because that charm is generated based on the upstream CF packages.
>
> Kapil, the problem I see with using rsyslog-forwarder is that the LTC
> doesn't send log messages to the syslog on its unit, but instead
> listens on the outgoing_port for a logdrain service to connect and
> then sends the log messages to that.  Perhaps the cf-logdrain charm
> could be nothing more than a service that connects to the LTC and
> dumps everything to syslog, and then we could connect the
> rsyslog-forwarder subordinate to that?  Would that be simpler than a
> service that just connects to two ports and sends all data from one of
> those ports to the other?
>
>
I think loggregator drain should go straight to logstash (tcp input and
formatter), i wanted to make the point of rsyslog forwarder as a general
statement about charms not having *nor* needing a syslog rel. If paas
infrastructure logging and viz needed, then forwarder subordinate related
to rsyslog aggregator can then be connected to logstash.. ie. lots of
pieces generically assembled as per orchestrator or user pref. else cf app
logging per loggregator to logstash via loggregator.

-k



> On Fri, Sep 5, 2014 at 2:26 PM, Kapil Thangavelu
> <kapil.thangavelu@xxxxxxxxxxxxx> wrote:
> > for existing charms you don't need one the syslog rel... you do
> > rsyslog-forwarder as a subordinate charm.
> >
> >
> > On Fri, Sep 5, 2014 at 2:17 PM, Alexander Lomov <lomov.as@xxxxxxxxx>
> wrote:
> >>
> >> Agree with you, Cory. Thank you for investigating this question.
> >>
> >> I've mentioned that there is no appropriate syslog relation in existing
> >> charms (for rsyslog and logstash). That's why I was planning to update
> this
> >> charms. By the way, do you think it would be useful to make them work
> with
> >> latest version of charm helpers (I mean services and contexts helpers)?
> >>
> >> Also I think that it would be a good idea to have one common relation to
> >> connect log aggregators with log emitters. In order to do it I will
> need to
> >> add new relation called syslog-drain to loggregator_trafficcontroller.
> >>
> >> The fact that loggregator_trafficcontroller relations exports data for
> >> syslog protocol is very useful (I thought it works through protobuf).
> >>
> >> Best wishes,
> >> Alex L.
> >>
> >>
> >>
> >> On 5 September 2014 20:46, Cory Johns <cory.johns@xxxxxxxxxxxxx> wrote:
> >>>
> >>> I was taking a closer look at the rsyslog and logstash charms to get a
> >>> better idea of how the CF log collector charm will work, and I wanted
> >>> to share some notes.  I also added Charles to this thread because he
> >>> has done some work on the rsyslog charm, and I thought he could make
> >>> sure my assumptions about how it works are valid.
> >>>
> >>> First, loggregator_trafficcontrol listens on
> >>> "loggregator_trafficcontroller" interface, using the "host" and
> >>> "endpoint_port" fields to provide connection information, and then
> >>> emits logs using the syslog protocol defined in RFC 5424 to whatever
> >>> connects to it.
> >>>
> >>> Second, the rsyslog seems to listen on the "syslog" interface, but
> >>> doesn't provide anything on that interface other than the implicit
> >>> "private-address" field and just assumes the standard port of 514 on
> >>> both TCP and UDP.  The default protocol appears to be RFC 5424.
> >>> [Charles: Is all of this correct?]
> >>>
> >>> Third, the logstash-indexer charm doesn't appear to support the syslog
> >>> (RFC 5424) procotol, but it should be fairly easy to add it to the
> >>> charm using the input config given at [1], below.  The config block is
> >>> a little more complex because it apparently does translation between
> >>> RFC 3164 and RFC 5424, but it seems like it should just be copy &
> >>> pasting that into the logstash-indexer charm.  Regarding the
> >>> logstash-agent charm, I'm certain we don't need it at all.
> >>>
> >>> Fourth, we don't want to have to push a CF app to do the log drain, so
> >>> we'll want an equivalent service in the CF log collector charm that
> >>> connects to the "host" and "endpoint_port" fields from the
> >>> "loggregator_trafficcontroller" relation to do the draining.  This
> >>> part I'm less clear about, but it seems like it should just be a basic
> >>> transparent proxy?  It will have to be a "male-to-male" proxy, though,
> >>> because both the LTC and the other end (rsyslog or logstash-indexer)
> >>> both expect to be connected to, even though the LTC side then emits to
> >>> whatever connects to it while the other end consumes.  So, the proxy
> >>> has to do the connecting on both sides.
> >>>
> >>>
> >>> [1]:
> >>>
> http://scottfrederick.cfapps.io/blog/2014/02/20/cloud-foundry-and-logstash
> >>
> >>
> >>
> >> --
> >> Mailing list: https://launchpad.net/~cf-charmers
> >> Post to     : cf-charmers@xxxxxxxxxxxxxxxxxxx
> >> Unsubscribe : https://launchpad.net/~cf-charmers
> >> More help   : https://help.launchpad.net/ListHelp
> >>
> >
>