cf-charmers team mailing list archive

[Alexander Lomov <lomov.as@xxxxxxxxx>]

Hey, Kapil.

Do you mean the solution is to create two different specific relations for
rsyslog-forwarder and logstash-agent charm (as I understand it works as
logstash shipper agent and should be connected through redis relation [1])?

I've checked rsyslog-forwarder, it seems to have what we need. From first
sight I was confused that rsyslog charm didn't have aggregation relation
hook. One left question for me is this about `juju-info` relation [2].
Could you tell what it is about?

And question out of the topic, about Juju GUI. Bundles have references to
charm repositories in following format: cs:precise/logstash-indexer-2.
Could you tell what does the last number mean?


[1]:
http://bazaar.launchpad.net/~charmers/charms/precise/logstash-agent/trunk/view/head:/metadata.yaml#L13
[2]:
http://bazaar.launchpad.net/~charmers/charms/precise/rsyslog-forwarder/trunk/view/head:/metadata.yaml#L8

Best wishes,
Alex L.


On 5 September 2014 22:18, Kapil Thangavelu <kapil.thangavelu@xxxxxxxxxxxxx>
wrote:

>
>
>
> On Fri, Sep 5, 2014 at 3:06 PM, Cory Johns <cory.johns@xxxxxxxxxxxxx>
> wrote:
>
>> Alexander,
>>
>> We should not need a new interface on the existing charms.  We will
>> need a service to perform the log draining (as I noted in my last
>> point), which should essentially adapt the existing
>> "loggregator_trafficcontroller" interface to the "syslog" interface.
>> I think this would be best to be its own charm, e.g. cf-logdrain, and
>> then should be able to connect CF to anything that supports the
>> "syslog" interface.  My other proposal was to add support for this
>> interface to the logstash-indexer charm.
>>
>> The reason cf-logdrain should be its own charm and that we should
>> avoid making changes to the loggregator_trafficcontrol charm is
>> because that charm is generated based on the upstream CF packages.
>>
>> Kapil, the problem I see with using rsyslog-forwarder is that the LTC
>> doesn't send log messages to the syslog on its unit, but instead
>> listens on the outgoing_port for a logdrain service to connect and
>> then sends the log messages to that.  Perhaps the cf-logdrain charm
>> could be nothing more than a service that connects to the LTC and
>> dumps everything to syslog, and then we could connect the
>> rsyslog-forwarder subordinate to that?  Would that be simpler than a
>> service that just connects to two ports and sends all data from one of
>> those ports to the other?
>>
>>
> I think loggregator drain should go straight to logstash (tcp input and
> formatter), i wanted to make the point of rsyslog forwarder as a general
> statement about charms not having *nor* needing a syslog rel. If paas
> infrastructure logging and viz needed, then forwarder subordinate related
> to rsyslog aggregator can then be connected to logstash.. ie. lots of
> pieces generically assembled as per orchestrator or user pref. else cf app
> logging per loggregator to logstash via loggregator.
>
> -k
>
>
>
>> On Fri, Sep 5, 2014 at 2:26 PM, Kapil Thangavelu
>> <kapil.thangavelu@xxxxxxxxxxxxx> wrote:
>> > for existing charms you don't need one the syslog rel... you do
>> > rsyslog-forwarder as a subordinate charm.
>> >
>> >
>> > On Fri, Sep 5, 2014 at 2:17 PM, Alexander Lomov <lomov.as@xxxxxxxxx>
>> wrote:
>> >>
>> >> Agree with you, Cory. Thank you for investigating this question.
>> >>
>> >> I've mentioned that there is no appropriate syslog relation in existing
>> >> charms (for rsyslog and logstash). That's why I was planning to update
>> this
>> >> charms. By the way, do you think it would be useful to make them work
>> with
>> >> latest version of charm helpers (I mean services and contexts helpers)?
>> >>
>> >> Also I think that it would be a good idea to have one common relation
>> to
>> >> connect log aggregators with log emitters. In order to do it I will
>> need to
>> >> add new relation called syslog-drain to loggregator_trafficcontroller.
>> >>
>> >> The fact that loggregator_trafficcontroller relations exports data for
>> >> syslog protocol is very useful (I thought it works through protobuf).
>> >>
>> >> Best wishes,
>> >> Alex L.
>> >>
>> >>
>> >>
>> >> On 5 September 2014 20:46, Cory Johns <cory.johns@xxxxxxxxxxxxx>
>> wrote:
>> >>>
>> >>> I was taking a closer look at the rsyslog and logstash charms to get a
>> >>> better idea of how the CF log collector charm will work, and I wanted
>> >>> to share some notes.  I also added Charles to this thread because he
>> >>> has done some work on the rsyslog charm, and I thought he could make
>> >>> sure my assumptions about how it works are valid.
>> >>>
>> >>> First, loggregator_trafficcontrol listens on
>> >>> "loggregator_trafficcontroller" interface, using the "host" and
>> >>> "endpoint_port" fields to provide connection information, and then
>> >>> emits logs using the syslog protocol defined in RFC 5424 to whatever
>> >>> connects to it.
>> >>>
>> >>> Second, the rsyslog seems to listen on the "syslog" interface, but
>> >>> doesn't provide anything on that interface other than the implicit
>> >>> "private-address" field and just assumes the standard port of 514 on
>> >>> both TCP and UDP.  The default protocol appears to be RFC 5424.
>> >>> [Charles: Is all of this correct?]
>> >>>
>> >>> Third, the logstash-indexer charm doesn't appear to support the syslog
>> >>> (RFC 5424) procotol, but it should be fairly easy to add it to the
>> >>> charm using the input config given at [1], below.  The config block is
>> >>> a little more complex because it apparently does translation between
>> >>> RFC 3164 and RFC 5424, but it seems like it should just be copy &
>> >>> pasting that into the logstash-indexer charm.  Regarding the
>> >>> logstash-agent charm, I'm certain we don't need it at all.
>> >>>
>> >>> Fourth, we don't want to have to push a CF app to do the log drain, so
>> >>> we'll want an equivalent service in the CF log collector charm that
>> >>> connects to the "host" and "endpoint_port" fields from the
>> >>> "loggregator_trafficcontroller" relation to do the draining.  This
>> >>> part I'm less clear about, but it seems like it should just be a basic
>> >>> transparent proxy?  It will have to be a "male-to-male" proxy, though,
>> >>> because both the LTC and the other end (rsyslog or logstash-indexer)
>> >>> both expect to be connected to, even though the LTC side then emits to
>> >>> whatever connects to it while the other end consumes.  So, the proxy
>> >>> has to do the connecting on both sides.
>> >>>
>> >>>
>> >>> [1]:
>> >>>
>> http://scottfrederick.cfapps.io/blog/2014/02/20/cloud-foundry-and-logstash
>> >>
>> >>
>> >>
>> >> --
>> >> Mailing list: https://launchpad.net/~cf-charmers
>> >> Post to     : cf-charmers@xxxxxxxxxxxxxxxxxxx
>> >> Unsubscribe : https://launchpad.net/~cf-charmers
>> >> More help   : https://help.launchpad.net/ListHelp
>> >>
>> >
>>
>
>