[syslog-ng] Syslog-ng input for beats ?

Scot scotrn at gmail.com
Fri May 12 01:21:42 UTC 2017 

Found this Gem!
https://www.balabit.com/documents/syslog-ng-ose-3.9-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#json-parser-options
Example 12.8. Convert logstash eventlog format v0 to v1

The following parser converts messages in the logstash eventlog v0 format
to the v1 format

parser p_jsoneventv0 {
  channel {
    parser { json-parser(extract-prefix("@fields")); };
    parser { json-parser(prefix(".json.")); };
    rewrite {
      set("1" value("@version"));
      set("${.json. at timestamp}" value("@timestamp"));
      set("${.json. at message}" value("message"));
    };
  };};

Added parser and updated log statement.
source s_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));};
destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"
template("$(format-json --scope dot-nv-pairs)\n"));};
log { source(s_BEATS);  parser(p_jsoneventv0); destination (d_jfile); };



[2017-05-11T21:17:19.433352] Incoming log entry;
line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":6000},"monitor":"http@
http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":"
http://192.168.1.16:9200
","tags":["beats_input_raw_event"],"duration":{"us":8000},"@timestamp":"2017-05-12T01:17:23.258Z","rtt":{"us":8000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}'
[2017-05-11T21:17:19.433471] *Error extracting JSON members into LogMessage
as the top-level JSON object is not an objec*t;
input='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":6000},"monitor":"http@
http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":"
http://192.168.1.16:9200
","tags":["beats_input_raw_event"],"duration":{"us":8000},"@timestamp":"2017-05-12T01:17:23.258Z","rtt":{"us":8000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}'
[2017-05-11T21:17:19.433495] Message parsing complete; result='0',
rule='p_jsoneventv0', location='/etc/syslog-ng/syslog-ng.conf:18:14'


On Thu, May 11, 2017 at 8:51 PM, Scot <scotrn at gmail.com> wrote:

> I'm almost there I think!
> An idea why is outputting message ={} nothing ?  What is
> rule='#anon-parser0
>
> *relative lines from syslog-ng.conf*
> source s_BEATS {network(port(*5140*) log-msg-size(65536)
> flags(no-parse));};
> destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"
> template("$(format-json --scope dot-nv-pairs)\n"));};
> log { source(s_BEATS);  parser {json-parser();}; destination (d_jfile); };
>
> */etc/logstash/conf.d/logstash.conf*
> input {
>   beats {
>     port => 5044
>   }
> }
>
> output {
>   tcp {
>     host => "192.168.1.16"
>     port => "*5140*"
>     mode => "client"
>   }
>
>
> *Running syslog-ng in foreground.  *
>
> [2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88',
> client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)'
> [2017-05-11T20:38:49.899997] Incoming log entry;
> line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_
> rtt":{"us":2000},"monitor":"http at http://192.168.1.16:9200"
> ,"type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200
> ","tags":["beats_input_raw_event"],"duration":{
> "us":5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"
> us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-
> LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":
> "1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"
> 192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@
> http://192.168.1.16:9200","type":"http","http_rtt":{"us":5000},"url":"
> http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{
> "us":7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:
> 42.020Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-
> LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":
> "1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"
> 192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@
> http://192.168.1.16:9200","type":"'
>
> [2017-05-11T20:38:49.900179] Message parsing complete; result='1',
> rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33'
> [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88',
> client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)'
> [2017-05-11T20:38:49.900384] Outgoing message; message='{}
>
> On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel at uvic.ca> wrote:
>
>> Since you already have the no-parse flag on the source, everything goes
>> into $MESSAGE
>>
>> Make a file destination with the template of "$MESSAGE\n" and it should
>> contain the entire payload.
>>
>> Evan.
>>
>>
>> On 05/10/2017 12:57 PM, Scot wrote:
>>
>> Thanks Evan,
>>
>> Bumped it up to 32768
>>
>>  Error extracting JSON members into LogMessage as the top-level JSON
>> object is not an object; input='":"A
>> I think there may be something else I need to do with the payload.
>>
>> How would I dump everything to a file to look at it ?
>>
>>
>>
>>
>> On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>
>>> looks like you might be running into the maximum message size.
>>> Try setting the syslog-ng configuration item
>>>
>>> log_msg_size(64K);
>>>
>>>
>>>
>>> On 05/10/2017 10:50 AM, Scot wrote:
>>>
>>> Using a RAW TCP seems to be loosing some of the beats header data and
>>> messages are getting concatenated.
>>> Trying different options but I'm fumbling.
>>>
>>>   syslog-ng[4596]: Unparsable JSON stream encountered;
>>> input='=net"},"message":"Synchronization of a replica of an Active
>>> Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS
>>> Settings,CN=...blaaa"
>>>
>>>
>>> source s_BEATS          {network(port(5140) flags(no-parse));}
>>> parser p_json {
>>>     json-parser (prefix(".json."));
>>> };
>>> log { source(s_BEATS);  parser(p_json); destination (d_file); };
>>>
>>>
>>> Anyone have a howto or blog for using syslog-ng with json inputs ?
>>> I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all
>>> the input output and parser requirements together.
>>>
>>> Trying to get here
>>> winlogbeat->syslog-ng->ES
>>> winlogbeat->syslog-ng->SPLUNKForwader
>>> winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
>>>
>>> or
>>> winlogbeat->logstash->syslog-ng->ES
>>> ...
>>>
>>> On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli at in2p3.fr> wrote:
>>>
>>>> Hi,
>>>>
>>>> On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
>>>> > I'm trying to find a solution that will let me mirror my beats data
>>>> like
>>>> > syslog-ng lets me do with syslog traffic.
>>>>
>>>> As far as I know those tools simply send the data over TCP in JSON
>>>> format.
>>>> If you just need to do routing using syslog-ng, you can simply use
>>>> network
>>>> source with flags(no-parse). If you need to process the data using
>>>> syslog-ng, you'll also need the json-parser().
>>>>
>>>> Cheers
>>>>
>>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170511/9e56ccff/attachment-0001.html>

More information about the syslog-ng mailing list