[syslog-ng] Syslog-ng input for beats ?

Fri May 12 01:28:58 UTC 2017

What ever is feeding this source appears to be truncating the message to 
1024 characters. I assume it is some kind of syslog stream because 1024 
is the syslog limit for entire message size.

I think you need to adjust your configuration of the originating software.

Evan.

On 05/11/2017 05:51 PM, Scot wrote:
> I'm almost there I think!
> An idea why is outputting message ={} nothing ?  What is 
> rule='#anon-parser0
>
> *relative lines from syslog-ng.conf*
> source s_BEATS{network(port(*5140*) log-msg-size(65536) 
> flags(no-parse));};
> destination d_jfile { 
> file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" 
> template("$(format-json --scope dot-nv-pairs)\n"));};
> log { source(s_BEATS);  parser {json-parser();}; destination (d_jfile); };
>
> *//etc/logstash/conf.d/logstash.conf/*
> input {
>   beats {
>     port => 5044
>   }
> }
>
> output {
>   tcp {
>     host => "192.168.1.16"
>     port => "*5140*"
>     mode => "client"
>   }
>
>
> *_Running syslog-ng in foreground. _*
>
> [2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', 
> client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140 
> <http://0.0.0.0:5140>)'
> [2017-05-11T20:38:49.899997] Incoming log entry; 
> line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http at http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{"us":5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http at http://192.168.1.16:9200","type":"http","http_rtt":{"us":5000},"url":"http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{"us":7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:42.020Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http at http://192.168.1.16:9200","type":"'
>
> [2017-05-11T20:38:49.900179] Message parsing complete; result='1', 
> rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33'
> [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', 
> client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140 
> <http://0.0.0.0:5140>)'
> [2017-05-11T20:38:49.900384] Outgoing message; message='{}
>
> On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel at uvic.ca 
> <mailto:erempel at uvic.ca>> wrote:
>
>     Since you already have the no-parse flag on the source, everything
>     goes into $MESSAGE
>
>     Make a file destination with the template of "$MESSAGE\n" and it
>     should contain the entire payload.
>
>     Evan.
>
>
>     On 05/10/2017 12:57 PM, Scot wrote:
>>     Thanks Evan,
>>
>>     Bumped it up to 32768
>>
>>      Error extracting JSON members into LogMessage as the top-level
>>     JSON object is not an object; input='":"A
>>     I think there may be something else I need to do with the payload.
>>
>>     How would I dump everything to a file to look at it ?
>>
>>
>>
>>
>>     On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel at uvic.ca
>>     <mailto:erempel at uvic.ca>> wrote:
>>
>>         looks like you might be running into the maximum message size.
>>         Try setting the syslog-ng configuration item
>>
>>         log_msg_size(64K);
>>
>>
>>
>>         On 05/10/2017 10:50 AM, Scot wrote:
>>>         Using a RAW TCP seems to be loosing some of the beats header
>>>         data and messages are getting concatenated.
>>>         Trying different options but I'm fumbling.
>>>
>>>           syslog-ng[4596]: Unparsable JSON stream encountered;
>>>         input='=net"},"message":"Synchronization of a replica of an
>>>         Active Directory naming context has begun.\n\nDestination
>>>         DRA:\tCN=NTDS Settings,CN=...blaaa"
>>>
>>>
>>>         source s_BEATS  {network(port(5140) flags(no-parse));}
>>>         parser p_json {
>>>             json-parser (prefix(".json."));
>>>         };
>>>         log { source(s_BEATS);  parser(p_json); destination (d_file); };
>>>
>>>
>>>         Anyone have a howto or blog for using syslog-ng with json
>>>         inputs ?
>>>         I'm looking at the syslog-ng-ose-latest-guides but it's hard
>>>         to put all the input output and parser requirements together.
>>>
>>>         Trying to get here
>>>         winlogbeat->syslog-ng->ES
>>>         winlogbeat->syslog-ng->SPLUNKForwader
>>>         winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
>>>
>>>         or
>>>         winlogbeat->logstash->syslog-ng->ES
>>>         ...
>>>
>>>         On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli
>>>         <wernli at in2p3.fr <mailto:wernli at in2p3.fr>> wrote:
>>>
>>>             Hi,
>>>
>>>             On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
>>>             > I'm trying to find a solution that will let me mirror
>>>             my beats data like
>>>             > syslog-ng lets me do with syslog traffic.
>>>
>>>             As far as I know those tools simply send the data over
>>>             TCP in JSON format.
>>>             If you just need to do routing using syslog-ng, you can
>>>             simply use network
>>>             source with flags(no-parse). If you need to process the
>>>             data using
>>>             syslog-ng, you'll also need the json-parser().
>>>
>>>             Cheers
>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170511/2efce9e5/attachment.html>