cf-charmers team mailing list archive

[Cory Johns <cory.johns@xxxxxxxxxxxxx>]

I was taking a closer look at the rsyslog and logstash charms to get a
better idea of how the CF log collector charm will work, and I wanted
to share some notes.  I also added Charles to this thread because he
has done some work on the rsyslog charm, and I thought he could make
sure my assumptions about how it works are valid.

First, loggregator_trafficcontrol listens on
"loggregator_trafficcontroller" interface, using the "host" and
"endpoint_port" fields to provide connection information, and then
emits logs using the syslog protocol defined in RFC 5424 to whatever
connects to it.

Second, the rsyslog seems to listen on the "syslog" interface, but
doesn't provide anything on that interface other than the implicit
"private-address" field and just assumes the standard port of 514 on
both TCP and UDP.  The default protocol appears to be RFC 5424.
[Charles: Is all of this correct?]

Third, the logstash-indexer charm doesn't appear to support the syslog
(RFC 5424) procotol, but it should be fairly easy to add it to the
charm using the input config given at [1], below.  The config block is
a little more complex because it apparently does translation between
RFC 3164 and RFC 5424, but it seems like it should just be copy &
pasting that into the logstash-indexer charm.  Regarding the
logstash-agent charm, I'm certain we don't need it at all.

Fourth, we don't want to have to push a CF app to do the log drain, so
we'll want an equivalent service in the CF log collector charm that
connects to the "host" and "endpoint_port" fields from the
"loggregator_trafficcontroller" relation to do the draining.  This
part I'm less clear about, but it seems like it should just be a basic
transparent proxy?  It will have to be a "male-to-male" proxy, though,
because both the LTC and the other end (rsyslog or logstash-indexer)
both expect to be connected to, even though the LTC side then emits to
whatever connects to it while the other end consumes.  So, the proxy
has to do the connecting on both sides.


[1]: http://scottfrederick.cfapps.io/blog/2014/02/20/cloud-foundry-and-logstash