[syslog-ng] Syslog-ng input for beats ? [SUMMARY01]

[Scot]

Hi All,

Just wanted to report my progress on this subject, and say thanks for your
input.

 I'm pretty happy with this setup so far I'll try to keep it short but
wanted to share.

*tl;dnr*
HOWTO send Windows log data through syslog-ng and get it perfectly parsed
in splunk and elasticsearch.

*Purpose*:
Funnel Windows events from enterprise or dmz zones through an remote
aggregation relay into syslog-ng, allowing syslog-ng to replicate and
broker log data based on business needs.

*Components*:
*Beats*- Winlogbeat is one of the beats agents by Elastic.co
(Elasticsearch) that run on windows hosts to offload events to Logstash in
this case.
*Logstash*: Like beats it is able to take plugin based input and convert it
to a plugin based output. I use it like a syslog relay but for compressed
json streams.
*syslog-ng:* Primary data aggregation and rule based distribution to
SPLUNK,ES and Archive file.

Beats support plugin based outputs, the primary two are es and logstash.
This solution uses a logstash instance running on the same VM as my rsyslog
relay. The output from winlogbeat, running on all windows servers, is a
compressed json data stream to its designated logstash relay IP:PORT.

This looks like this
winlogbeat
winlogbeat
winlogbeat ---> TCP:5044 --> Logstash --> TCP:9150 -->  syslog-ng
winlogbeat
winlogbeat

Systems local to the syslog-ng server also go through a logstash hop.
Logstash ads a single aggregator tag to each json message and can also take
rule based actions like VictorOps, or massage data inline.

syslog-ng then directs a copy of the json stream to a splunk forwarder,
archive file, elasticsearch and any other output supported by Balabit.

*Advantage*: I can write rules to tier my data to best of class or lower
cost data analytic solutions or even new ones with a syslog-ng.

*sylog-ng.conf *
# This listens for data from logstash output.
source s_BEATS          {network(port(9150) log-msg-size(65536)
flags(no-parse));};

# Define the output to splunkforwarder.
destination d_splunk_BEAT  {network("splunkforwarderhost"
        template("$(format-json --scope nv-pairs -p @timestamp=${ISODATE}
-x MESSAGE)\n") port(5140));};

# Send a copy to SPLUNK or filter f_splunk_beat
log { source(s_BEATS);  parser {json-parser();};
 destination(d_splunk_BEAT); };


*logstash beats-pipeline.conf output to syslog-ng*
output {
  tcp {
    host => "syslog-ng host"
    port => "9150"
    mode => "client"
    codec => "json_lines"
  }
}
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170526/2060ad3f/attachment.html>