[syslog-ng] Syslog-ng input for beats ?

Fri May 12 01:55:31 UTC 2017

Forgot to post the debut output. How would I remove the "time and host"
prefix added by syslog-ng to the output ?

[2017-05-11T21:53:19.452143] Incoming log entry;
line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@
http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":"
http://192.168.1.16:9200
","tags":["beats_input_raw_event"],"duration":{"us":4000},"rtt":{"us":4000},"@timestamp":"2017-05-12T01:53:23.258Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}'

[2017-05-11T21:53:19.452218] Message parsing complete; result='1'

[2017-05-11T21:53:19.452364] Outgoing message; message=*'May 11 21:53:19
hal*
{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@
http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":"
http://192.168.1.16:9200
","tags":["beats_input_raw_event"],"duration":{"us":4000},"rtt":{"us":4000},"@timestamp":"2017-05-12T01:53:23.258Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}

On Thu, May 11, 2017 at 9:52 PM, Scot <scotrn at gmail.com> wrote:

> I think it's working with kv-parser!!
> #
> source s_BEATS          {network(port(5140) log-msg-size(65536)
> flags(no-parse));};
> #
> # Gives Outgoing message; message='{}
> # destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"
> template("$(format-json --scope dot-nv-pairs)\n"));};
> #
> destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json");
> };
> #
> #
> log { source(s_BEATS);  parser {kv-parser();}; destination (d_jfile); };
>
>
> On Thu, May 11, 2017 at 9:44 PM, Scot <scotrn at gmail.com> wrote:
>
>> Comparing the output from running syslog in --debug -F  I can see the
>> payload of the json before and after parsing.  They look the same.  I think
>> it's the json parsing on the receiving side. I added an option to logstash
>> to force one line per event.  codec => "json_lines"
>>
>> This is the output from logstash writing to local file.
>> {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us"
>> :5000},"monitor":"http at http://192.168.1.16:9200","type":"htt
>> p","http_rtt":{"us":8000},"url":"http://192.168.1.16:9200","
>> tags":["beats_input_raw_event"],"duration":{"us":14000},"@
>> timestamp":"2017-05-12T01:32:13.258Z","rtt":{"us":14000},"
>> port":9200,"response":{"status":200},"beat":{"hostname
>> ":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.
>> 0"},"@version":"1","host":"TYLER-LAPTOP","up":true}
>>
>>
>> This is the stdout from syslog-ng.  Looks like everything is there its
>> just in there parsing.
>> *Incoming log entry; line='*
>> {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us"
>> :5000},"monitor":"http at http://192.168.1.16:9200","type":"htt
>> p","http_rtt":{"us":4000},"url":"http://192.168.1.16:9200","
>> tags":["beats_input_raw_event"],"duration":{"us":9000},"rtt"
>> :{"us":9000},"@timestamp":"2017-05-12T01:31:39.258Z","
>> port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","
>> name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","
>> host":"TYLER-LAPTOP","up":true}'
>>
>>
>> On Thu, May 11, 2017 at 9:28 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>
>>> What ever is feeding this source appears to be truncating the message to
>>> 1024 characters. I assume it is some kind of syslog stream because 1024 is
>>> the syslog limit for entire message size.
>>>
>>> I think you need to adjust your configuration of the originating
>>> software.
>>>
>>> Evan.
>>>
>>>
>>>
>>> On 05/11/2017 05:51 PM, Scot wrote:
>>>
>>> I'm almost there I think!
>>> An idea why is outputting message ={} nothing ?  What is
>>> rule='#anon-parser0
>>>
>>> *relative lines from syslog-ng.conf*
>>> source s_BEATS {network(port(*5140*) log-msg-size(65536)
>>> flags(no-parse));};
>>> destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"
>>> template("$(format-json --scope dot-nv-pairs)\n"));};
>>> log { source(s_BEATS);  parser {json-parser();}; destination (d_jfile);
>>> };
>>>
>>> */etc/logstash/conf.d/logstash.conf*
>>> input {
>>>   beats {
>>>     port => 5044
>>>   }
>>> }
>>>
>>> output {
>>>   tcp {
>>>     host => "192.168.1.16"
>>>     port => "*5140*"
>>>     mode => "client"
>>>   }
>>>
>>>
>>> *Running syslog-ng in foreground.  *
>>>
>>> [2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88',
>>> client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)'
>>> [2017-05-11T20:38:49.899997] Incoming log entry;
>>> line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt"
>>> :{"us":2000},"monitor":"http at http://192.168.1.16:9200","type
>>> ":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200
>>> ","tags":["beats_input_raw_event"],"duration":{"us":5000
>>> },"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us":5000},
>>> "port":9200,"response":{"status":200},"beat":{"hostname
>>> ":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.
>>> 0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"
>>> scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":
>>> 2000},"monitor":"http at http://192.168.1.16:9200","type":"
>>> http","http_rtt":{"us":5000},"url":"http://192.168.1.16:9200
>>> ","tags":["beats_input_raw_event"],"duration":{"us":7000}
>>> ,"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:42.020Z","
>>> port":9200,"response":{"status":200},"beat":{"hostname
>>> ":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.
>>> 0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"
>>> scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":
>>> 2000},"monitor":"http at http://192.168.1.16:9200","type":"'
>>>
>>> [2017-05-11T20:38:49.900179] Message parsing complete; result='1',
>>> rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33'
>>> [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88',
>>> client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)'
>>> [2017-05-11T20:38:49.900384] Outgoing message; message='{}
>>>
>>> On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>>
>>>> Since you already have the no-parse flag on the source, everything goes
>>>> into $MESSAGE
>>>>
>>>> Make a file destination with the template of "$MESSAGE\n" and it should
>>>> contain the entire payload.
>>>>
>>>> Evan.
>>>>
>>>>
>>>> On 05/10/2017 12:57 PM, Scot wrote:
>>>>
>>>> Thanks Evan,
>>>>
>>>> Bumped it up to 32768
>>>>
>>>>  Error extracting JSON members into LogMessage as the top-level JSON
>>>> object is not an object; input='":"A
>>>> I think there may be something else I need to do with the payload.
>>>>
>>>> How would I dump everything to a file to look at it ?
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel at uvic.ca> wrote:
>>>>
>>>>> looks like you might be running into the maximum message size.
>>>>> Try setting the syslog-ng configuration item
>>>>>
>>>>> log_msg_size(64K);
>>>>>
>>>>>
>>>>>
>>>>> On 05/10/2017 10:50 AM, Scot wrote:
>>>>>
>>>>> Using a RAW TCP seems to be loosing some of the beats header data and
>>>>> messages are getting concatenated.
>>>>> Trying different options but I'm fumbling.
>>>>>
>>>>>   syslog-ng[4596]: Unparsable JSON stream encountered;
>>>>> input='=net"},"message":"Synchronization of a replica of an Active
>>>>> Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS
>>>>> Settings,CN=...blaaa"
>>>>>
>>>>>
>>>>> source s_BEATS          {network(port(5140) flags(no-parse));}
>>>>> parser p_json {
>>>>>     json-parser (prefix(".json."));
>>>>> };
>>>>> log { source(s_BEATS);  parser(p_json); destination (d_file); };
>>>>>
>>>>>
>>>>> Anyone have a howto or blog for using syslog-ng with json inputs ?
>>>>> I'm looking at the syslog-ng-ose-latest-guides but it's hard to put
>>>>> all the input output and parser requirements together.
>>>>>
>>>>> Trying to get here
>>>>> winlogbeat->syslog-ng->ES
>>>>> winlogbeat->syslog-ng->SPLUNKForwader
>>>>> winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
>>>>>
>>>>> or
>>>>> winlogbeat->logstash->syslog-ng->ES
>>>>> ...
>>>>>
>>>>> On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli at in2p3.fr> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
>>>>>> > I'm trying to find a solution that will let me mirror my beats data
>>>>>> like
>>>>>> > syslog-ng lets me do with syslog traffic.
>>>>>>
>>>>>> As far as I know those tools simply send the data over TCP in JSON
>>>>>> format.
>>>>>> If you just need to do routing using syslog-ng, you can simply use
>>>>>> network
>>>>>> source with flags(no-parse). If you need to process the data using
>>>>>> syslog-ng, you'll also need the json-parser().
>>>>>>
>>>>>> Cheers
>>>>>>
>>>>>>
>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170511/b526f064/attachment-0001.html>