cf-charmers team mailing list archive

[Cory Johns <cory.johns@xxxxxxxxxxxxx>]

Alexander,

We should not need a new interface on the existing charms.  We will
need a service to perform the log draining (as I noted in my last
point), which should essentially adapt the existing
"loggregator_trafficcontroller" interface to the "syslog" interface.
I think this would be best to be its own charm, e.g. cf-logdrain, and
then should be able to connect CF to anything that supports the
"syslog" interface.  My other proposal was to add support for this
interface to the logstash-indexer charm.

The reason cf-logdrain should be its own charm and that we should
avoid making changes to the loggregator_trafficcontrol charm is
because that charm is generated based on the upstream CF packages.

Kapil, the problem I see with using rsyslog-forwarder is that the LTC
doesn't send log messages to the syslog on its unit, but instead
listens on the outgoing_port for a logdrain service to connect and
then sends the log messages to that.  Perhaps the cf-logdrain charm
could be nothing more than a service that connects to the LTC and
dumps everything to syslog, and then we could connect the
rsyslog-forwarder subordinate to that?  Would that be simpler than a
service that just connects to two ports and sends all data from one of
those ports to the other?

On Fri, Sep 5, 2014 at 2:26 PM, Kapil Thangavelu
<kapil.thangavelu@xxxxxxxxxxxxx> wrote:
> for existing charms you don't need one the syslog rel... you do
> rsyslog-forwarder as a subordinate charm.
>
>
> On Fri, Sep 5, 2014 at 2:17 PM, Alexander Lomov <lomov.as@xxxxxxxxx> wrote:
>>
>> Agree with you, Cory. Thank you for investigating this question.
>>
>> I've mentioned that there is no appropriate syslog relation in existing
>> charms (for rsyslog and logstash). That's why I was planning to update this
>> charms. By the way, do you think it would be useful to make them work with
>> latest version of charm helpers (I mean services and contexts helpers)?
>>
>> Also I think that it would be a good idea to have one common relation to
>> connect log aggregators with log emitters. In order to do it I will need to
>> add new relation called syslog-drain to loggregator_trafficcontroller.
>>
>> The fact that loggregator_trafficcontroller relations exports data for
>> syslog protocol is very useful (I thought it works through protobuf).
>>
>> Best wishes,
>> Alex L.
>>
>>
>>
>> On 5 September 2014 20:46, Cory Johns <cory.johns@xxxxxxxxxxxxx> wrote:
>>>
>>> I was taking a closer look at the rsyslog and logstash charms to get a
>>> better idea of how the CF log collector charm will work, and I wanted
>>> to share some notes.  I also added Charles to this thread because he
>>> has done some work on the rsyslog charm, and I thought he could make
>>> sure my assumptions about how it works are valid.
>>>
>>> First, loggregator_trafficcontrol listens on
>>> "loggregator_trafficcontroller" interface, using the "host" and
>>> "endpoint_port" fields to provide connection information, and then
>>> emits logs using the syslog protocol defined in RFC 5424 to whatever
>>> connects to it.
>>>
>>> Second, the rsyslog seems to listen on the "syslog" interface, but
>>> doesn't provide anything on that interface other than the implicit
>>> "private-address" field and just assumes the standard port of 514 on
>>> both TCP and UDP.  The default protocol appears to be RFC 5424.
>>> [Charles: Is all of this correct?]
>>>
>>> Third, the logstash-indexer charm doesn't appear to support the syslog
>>> (RFC 5424) procotol, but it should be fairly easy to add it to the
>>> charm using the input config given at [1], below.  The config block is
>>> a little more complex because it apparently does translation between
>>> RFC 3164 and RFC 5424, but it seems like it should just be copy &
>>> pasting that into the logstash-indexer charm.  Regarding the
>>> logstash-agent charm, I'm certain we don't need it at all.
>>>
>>> Fourth, we don't want to have to push a CF app to do the log drain, so
>>> we'll want an equivalent service in the CF log collector charm that
>>> connects to the "host" and "endpoint_port" fields from the
>>> "loggregator_trafficcontroller" relation to do the draining.  This
>>> part I'm less clear about, but it seems like it should just be a basic
>>> transparent proxy?  It will have to be a "male-to-male" proxy, though,
>>> because both the LTC and the other end (rsyslog or logstash-indexer)
>>> both expect to be connected to, even though the LTC side then emits to
>>> whatever connects to it while the other end consumes.  So, the proxy
>>> has to do the connecting on both sides.
>>>
>>>
>>> [1]:
>>> http://scottfrederick.cfapps.io/blog/2014/02/20/cloud-foundry-and-logstash
>>
>>
>>
>> --
>> Mailing list: https://launchpad.net/~cf-charmers
>> Post to     : cf-charmers@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~cf-charmers
>> More help   : https://help.launchpad.net/ListHelp
>>
>