I think it's working with kv-parser!! # source s_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));}; # # Gives Outgoing message; message='{} # destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; # destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"); }; # # log { source(s_BEATS); parser {kv-parser();}; destination (d_jfile); }; On Thu, May 11, 2017 at 9:44 PM, Scot <scotrn@gmail.com> wrote:
Comparing the output from running syslog in --debug -F I can see the payload of the json before and after parsing. They look the same. I think it's the json parsing on the receiving side. I added an option to logstash to force one line per event. codec => "json_lines"
This is the output from logstash writing to local file. {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{" us":5000},"monitor":"http@http://192.168.1.16:9200","type":" http","http_rtt":{"us":8000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us": 14000},"@timestamp":"2017-05-12T01:32:13.258Z","rtt":{"us": 14000},"port":9200,"response":{"status":200},"beat":{" hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version" :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}
This is the stdout from syslog-ng. Looks like everything is there its just in there parsing. *Incoming log entry; line='* {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{" us":5000},"monitor":"http@http://192.168.1.16:9200","type":" http","http_rtt":{"us":4000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":9000} ,"rtt":{"us":9000},"@timestamp":"2017-05-12T01:31:39.258Z","port":9200," response":{"status":200},"beat":{"hostname":"TYLER- LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version": "1","host":"TYLER-LAPTOP","up":true}'
On Thu, May 11, 2017 at 9:28 PM, Evan Rempel <erempel@uvic.ca> wrote:
What ever is feeding this source appears to be truncating the message to 1024 characters. I assume it is some kind of syslog stream because 1024 is the syslog limit for entire message size.
I think you need to adjust your configuration of the originating software.
Evan.
On 05/11/2017 05:51 PM, Scot wrote:
I'm almost there I think! An idea why is outputting message ={} nothing ? What is rule='#anon-parser0
*relative lines from syslog-ng.conf* source s_BEATS {network(port(*5140*) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };
*/etc/logstash/conf.d/logstash.conf* input { beats { port => 5044 } }
output { tcp { host => "192.168.1.16" port => "*5140*" mode => "client" }
*Running syslog-ng in foreground. *
[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt" :{"us":2000},"monitor":"http@http://192.168.1.16:9200"," type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us": 5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us": 5000},"port":9200,"response":{"status":200},"beat":{" hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version" :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up": true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt": {"us":2000},"monitor":"http@http://192.168.1.16:9200","type" :"http","http_rtt":{"us":5000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us": 7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:4 2.020Z","port":9200,"response":{"status":200},"beat":{" hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version" :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up": true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt": {"us":2000},"monitor":"http@http://192.168.1.16:9200","type":"'
[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33' [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.900384] Outgoing message; message='{}
On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca> wrote:
Since you already have the no-parse flag on the source, everything goes into $MESSAGE
Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.
Evan.
On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq