Syslog-ng input for beats ?
Hi Has anyone tried using syslog-ng inputs for logstash or beats data ? I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic. my syslog data goes to file, es and splunk. Need the same split for winlogbeat Winlogbeat goes to logstash then ES. Thanks for reading.
Hi, On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser(). Cheers
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling. syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa" source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); }; Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together. Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json or winlogbeat->logstash->syslog-ng->ES ... On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item log_msg_size(64K); On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr <mailto:wernli@in2p3.fr>> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote: > I'm trying to find a solution that will let me mirror my beats data like > syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 Data Centre Services, University Systems, University of Victoria
Thanks Evan, Bumped it up to 32768 Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload. How would I dump everything to a file to look at it ? On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Evan Rempel erempel@uvic.ca Senior Systems Administrator 250.721.7691 <(250)%20721-7691> Data Centre Services, University Systems, University of Victoria
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Since you already have the no-parse flag on the source, everything goes into $MESSAGE Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload. Evan. On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr <mailto:wernli@in2p3.fr>> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote: > I'm trying to find a solution that will let me mirror my beats data like > syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
I'm almost there I think! An idea why is outputting message ={} nothing ? What is rule='#anon-parser0 *relative lines from syslog-ng.conf* source s_BEATS {network(port(*5140*) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); }; */etc/logstash/conf.d/logstash.conf* input { beats { port => 5044 } } output { tcp { host => "192.168.1.16" port => "*5140*" mode => "client" } *Running syslog-ng in foreground. * [2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":5000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:42.020Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"' [2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33' [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.900384] Outgoing message; message='{} On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca> wrote:
Since you already have the no-parse flag on the source, everything goes into $MESSAGE
Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.
Evan.
On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Found this Gem! https://www.balabit.com/documents/syslog-ng-ose-3.9-guides/en/syslog-ng-ose-... Example 12.8. Convert logstash eventlog format v0 to v1 The following parser converts messages in the logstash eventlog v0 format to the v1 format parser p_jsoneventv0 { channel { parser { json-parser(extract-prefix("@fields")); }; parser { json-parser(prefix(".json.")); }; rewrite { set("1" value("@version")); set("${.json.@timestamp}" value("@timestamp")); set("${.json.@message}" value("message")); }; };}; Added parser and updated log statement. source s_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser(p_jsoneventv0); destination (d_jfile); }; [2017-05-11T21:17:19.433352] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":6000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":8000},"@timestamp":"2017-05-12T01:17:23.258Z","rtt":{"us":8000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}' [2017-05-11T21:17:19.433471] *Error extracting JSON members into LogMessage as the top-level JSON object is not an objec*t; input='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":6000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":8000},"@timestamp":"2017-05-12T01:17:23.258Z","rtt":{"us":8000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}' [2017-05-11T21:17:19.433495] Message parsing complete; result='0', rule='p_jsoneventv0', location='/etc/syslog-ng/syslog-ng.conf:18:14' On Thu, May 11, 2017 at 8:51 PM, Scot <scotrn@gmail.com> wrote:
I'm almost there I think! An idea why is outputting message ={} nothing ? What is rule='#anon-parser0
*relative lines from syslog-ng.conf* source s_BEATS {network(port(*5140*) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };
*/etc/logstash/conf.d/logstash.conf* input { beats { port => 5044 } }
output { tcp { host => "192.168.1.16" port => "*5140*" mode => "client" }
*Running syslog-ng in foreground. *
[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_ rtt":{"us":2000},"monitor":"http@http://192.168.1.16:9200" ,"type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{ "us":5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{" us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER- LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version": "1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":" 192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":5000},"url":" http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{ "us":7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30: 42.020Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER- LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version": "1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":" 192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"'
[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33' [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.900384] Outgoing message; message='{}
On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca> wrote:
Since you already have the no-parse flag on the source, everything goes into $MESSAGE
Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.
Evan.
On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
What ever is feeding this source appears to be truncating the message to 1024 characters. I assume it is some kind of syslog stream because 1024 is the syslog limit for entire message size. I think you need to adjust your configuration of the originating software. Evan. On 05/11/2017 05:51 PM, Scot wrote:
I'm almost there I think! An idea why is outputting message ={} nothing ? What is rule='#anon-parser0
*relative lines from syslog-ng.conf* source s_BEATS{network(port(*5140*) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };
*//etc/logstash/conf.d/logstash.conf/* input { beats { port => 5044 } }
output { tcp { host => "192.168.1.16" port => "*5140*" mode => "client" }
*_Running syslog-ng in foreground. _*
[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140 <http://0.0.0.0:5140>)' [2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{"us":5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@http://192.168.1.16:9200","type":"http","http_rtt":{"us":5000},"url":"http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{"us":7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:42.020Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@http://192.168.1.16:9200","type":"'
[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33' [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140 <http://0.0.0.0:5140>)' [2017-05-11T20:38:49.900384] Outgoing message; message='{}
On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
Since you already have the no-parse flag on the source, everything goes into $MESSAGE
Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.
Evan.
On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca <mailto:erempel@uvic.ca>> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr <mailto:wernli@in2p3.fr>> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote: > I'm trying to find a solution that will let me mirror my beats data like > syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
Comparing the output from running syslog in --debug -F I can see the payload of the json before and after parsing. They look the same. I think it's the json parsing on the receiving side. I added an option to logstash to force one line per event. codec => "json_lines" This is the output from logstash writing to local file. {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":5000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":8000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":14000},"@timestamp":"2017-05-12T01:32:13.258Z","rtt":{"us":14000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true} This is the stdout from syslog-ng. Looks like everything is there its just in there parsing. *Incoming log entry; line='* {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":5000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":4000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":9000},"rtt":{"us":9000},"@timestamp":"2017-05-12T01:31:39.258Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}' On Thu, May 11, 2017 at 9:28 PM, Evan Rempel <erempel@uvic.ca> wrote:
What ever is feeding this source appears to be truncating the message to 1024 characters. I assume it is some kind of syslog stream because 1024 is the syslog limit for entire message size.
I think you need to adjust your configuration of the originating software.
Evan.
On 05/11/2017 05:51 PM, Scot wrote:
I'm almost there I think! An idea why is outputting message ={} nothing ? What is rule='#anon-parser0
*relative lines from syslog-ng.conf* source s_BEATS {network(port(*5140*) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };
*/etc/logstash/conf.d/logstash.conf* input { beats { port => 5044 } }
output { tcp { host => "192.168.1.16" port => "*5140*" mode => "client" }
*Running syslog-ng in foreground. *
[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_ rtt":{"us":2000},"monitor":"http@http://192.168.1.16:9200" ,"type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{ "us":5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{" us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER- LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version": "1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":" 192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":5000},"url":" http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{ "us":7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30: 42.020Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER- LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version": "1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":" 192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"'
[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33' [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.900384] Outgoing message; message='{}
On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca> wrote:
Since you already have the no-parse flag on the source, everything goes into $MESSAGE
Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.
Evan.
On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I think it's working with kv-parser!! # source s_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));}; # # Gives Outgoing message; message='{} # destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; # destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"); }; # # log { source(s_BEATS); parser {kv-parser();}; destination (d_jfile); }; On Thu, May 11, 2017 at 9:44 PM, Scot <scotrn@gmail.com> wrote:
Comparing the output from running syslog in --debug -F I can see the payload of the json before and after parsing. They look the same. I think it's the json parsing on the receiving side. I added an option to logstash to force one line per event. codec => "json_lines"
This is the output from logstash writing to local file. {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{" us":5000},"monitor":"http@http://192.168.1.16:9200","type":" http","http_rtt":{"us":8000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us": 14000},"@timestamp":"2017-05-12T01:32:13.258Z","rtt":{"us": 14000},"port":9200,"response":{"status":200},"beat":{" hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version" :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}
This is the stdout from syslog-ng. Looks like everything is there its just in there parsing. *Incoming log entry; line='* {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{" us":5000},"monitor":"http@http://192.168.1.16:9200","type":" http","http_rtt":{"us":4000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":9000} ,"rtt":{"us":9000},"@timestamp":"2017-05-12T01:31:39.258Z","port":9200," response":{"status":200},"beat":{"hostname":"TYLER- LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version": "1","host":"TYLER-LAPTOP","up":true}'
On Thu, May 11, 2017 at 9:28 PM, Evan Rempel <erempel@uvic.ca> wrote:
What ever is feeding this source appears to be truncating the message to 1024 characters. I assume it is some kind of syslog stream because 1024 is the syslog limit for entire message size.
I think you need to adjust your configuration of the originating software.
Evan.
On 05/11/2017 05:51 PM, Scot wrote:
I'm almost there I think! An idea why is outputting message ={} nothing ? What is rule='#anon-parser0
*relative lines from syslog-ng.conf* source s_BEATS {network(port(*5140*) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };
*/etc/logstash/conf.d/logstash.conf* input { beats { port => 5044 } }
output { tcp { host => "192.168.1.16" port => "*5140*" mode => "client" }
*Running syslog-ng in foreground. *
[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt" :{"us":2000},"monitor":"http@http://192.168.1.16:9200"," type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us": 5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us": 5000},"port":9200,"response":{"status":200},"beat":{" hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version" :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up": true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt": {"us":2000},"monitor":"http@http://192.168.1.16:9200","type" :"http","http_rtt":{"us":5000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us": 7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:4 2.020Z","port":9200,"response":{"status":200},"beat":{" hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version" :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up": true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt": {"us":2000},"monitor":"http@http://192.168.1.16:9200","type":"'
[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33' [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.900384] Outgoing message; message='{}
On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca> wrote:
Since you already have the no-parse flag on the source, everything goes into $MESSAGE
Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.
Evan.
On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Forgot to post the debut output. How would I remove the "time and host" prefix added by syslog-ng to the output ? [2017-05-11T21:53:19.452143] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":4000},"rtt":{"us":4000},"@timestamp":"2017-05-12T01:53:23.258Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}' [2017-05-11T21:53:19.452218] Message parsing complete; result='1' [2017-05-11T21:53:19.452364] Outgoing message; message=*'May 11 21:53:19 hal* {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":4000},"rtt":{"us":4000},"@timestamp":"2017-05-12T01:53:23.258Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true} On Thu, May 11, 2017 at 9:52 PM, Scot <scotrn@gmail.com> wrote:
I think it's working with kv-parser!! # source s_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));}; # # Gives Outgoing message; message='{} # destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; # destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json"); }; # # log { source(s_BEATS); parser {kv-parser();}; destination (d_jfile); };
On Thu, May 11, 2017 at 9:44 PM, Scot <scotrn@gmail.com> wrote:
Comparing the output from running syslog in --debug -F I can see the payload of the json before and after parsing. They look the same. I think it's the json parsing on the receiving side. I added an option to logstash to force one line per event. codec => "json_lines"
This is the output from logstash writing to local file. {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us" :5000},"monitor":"http@http://192.168.1.16:9200","type":"htt p","http_rtt":{"us":8000},"url":"http://192.168.1.16:9200"," tags":["beats_input_raw_event"],"duration":{"us":14000},"@ timestamp":"2017-05-12T01:32:13.258Z","rtt":{"us":14000}," port":9200,"response":{"status":200},"beat":{"hostname ":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4. 0"},"@version":"1","host":"TYLER-LAPTOP","up":true}
This is the stdout from syslog-ng. Looks like everything is there its just in there parsing. *Incoming log entry; line='* {"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us" :5000},"monitor":"http@http://192.168.1.16:9200","type":"htt p","http_rtt":{"us":4000},"url":"http://192.168.1.16:9200"," tags":["beats_input_raw_event"],"duration":{"us":9000},"rtt" :{"us":9000},"@timestamp":"2017-05-12T01:31:39.258Z"," port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP"," name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1"," host":"TYLER-LAPTOP","up":true}'
On Thu, May 11, 2017 at 9:28 PM, Evan Rempel <erempel@uvic.ca> wrote:
What ever is feeding this source appears to be truncating the message to 1024 characters. I assume it is some kind of syslog stream because 1024 is the syslog limit for entire message size.
I think you need to adjust your configuration of the originating software.
Evan.
On 05/11/2017 05:51 PM, Scot wrote:
I'm almost there I think! An idea why is outputting message ={} nothing ? What is rule='#anon-parser0
*relative lines from syslog-ng.conf* source s_BEATS {network(port(*5140*) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };
*/etc/logstash/conf.d/logstash.conf* input { beats { port => 5044 } }
output { tcp { host => "192.168.1.16" port => "*5140*" mode => "client" }
*Running syslog-ng in foreground. *
[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt" :{"us":2000},"monitor":"http@http://192.168.1.16:9200","type ":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":5000 },"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us":5000}, "port":9200,"response":{"status":200},"beat":{"hostname ":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4. 0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{" scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us": 2000},"monitor":"http@http://192.168.1.16:9200","type":" http","http_rtt":{"us":5000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":7000} ,"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:42.020Z"," port":9200,"response":{"status":200},"beat":{"hostname ":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4. 0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{" scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us": 2000},"monitor":"http@http://192.168.1.16:9200","type":"'
[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33' [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.900384] Outgoing message; message='{}
On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca> wrote:
Since you already have the no-parse flag on the source, everything goes into $MESSAGE
Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.
Evan.
On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote: > I'm trying to find a solution that will let me mirror my beats data like > syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
To remove the time and host from the output you need to define a template to use with your destination. The template should be "$MESSAGE\n" On 05/11/2017 06:55 PM, Scot wrote:
Forgot to post the debut output. How would I remove the "time and host" prefix added by syslog-ng to the output ?
Yep, works great! Outgoing seems to add a \n just before the closing quote but much better. Now I should be able to have one json stream from my remote logstash server to syslog-ng broker. [2017-05-11T23:37:33.500956] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":5000},"rtt":{"us":4000},"@timestamp":"2017-05-12T03:37:37.258Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}' [2017-05-11T23:37:33.501033] Message parsing complete; result='1' [2017-05-11T23:37:33.501109] Outgoing message; message='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":" http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us":5000},"rtt":{"us":4000},"@timestamp":"2017-05-12T03:37:37.258Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true} ' On Thu, May 11, 2017 at 11:24 PM, Evan Rempel <erempel@uvic.ca> wrote:
To remove the time and host from the output you need to define a template to use with your destination. The template should be "$MESSAGE\n"
On 05/11/2017 06:55 PM, Scot wrote:
Forgot to post the debut output. How would I remove the "time and host" prefix added by syslog-ng to the output ?
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, YouTube specified --scope dot-nv-pairs tó format-json, which only includes name-value pairs that start with a dot. 2017. máj. 12. de. 2:51 ezt írta ("Scot" <scotrn@gmail.com>):
I'm almost there I think! An idea why is outputting message ={} nothing ? What is rule='#anon-parser0
*relative lines from syslog-ng.conf* source s_BEATS {network(port(*5140*) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };
*/etc/logstash/conf.d/logstash.conf* input { beats { port => 5044 } }
output { tcp { host => "192.168.1.16" port => "*5140*" mode => "client" }
*Running syslog-ng in foreground. *
[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_ rtt":{"us":2000},"monitor":"http@http://192.168.1.16:9200" ,"type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{ "us":5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{" us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER- LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version": "1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":" 192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"http","http_rtt":{"us":5000},"url":" http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{ "us":7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30: 42.020Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER- LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version": "1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":" 192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@ http://192.168.1.16:9200","type":"'
[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33' [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.900384] Outgoing message; message='{}
On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca> wrote:
Since you already have the no-parse flag on the source, everything goes into $MESSAGE
Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.
Evan.
On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Youtube ? Got the \n off the json log by just removing the \n from "$MESSAGE\n" Now that I have the json TCP stream from logstash into syslog-ng I want to spit it back out to multiple destinations. Sending this beats data back into elastic search is causing the message block to go into ES non parsed. How can I debug that ES output ? source s_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));}; destination d_es_beats { elasticsearch2( disk-buffer( reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option dir("/opt/syslog-ng/buffer") disk-buf-size(10485760) mem-buf-length(100000) # number of messages stored in overflow queue ) # END DiskBuffer client-mode("http") index("winlogbeat-${YEAR}.${MONTH}.${DAY}") type("winlogbeat") # Description: The type of the index. For example, type("test") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") #template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)") template("${MESSAGE}") cluster-url("http://192.168.1.16:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("hal") client_lib_dir("/usr/share/elasticsearch/lib") ); }; log { source(s_BEATS); parser {kv-parser();}; destination (d_es_beats); }; On Fri, May 12, 2017 at 12:09 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
Hi,
YouTube specified --scope dot-nv-pairs tó format-json, which only includes name-value pairs that start with a dot.
2017. máj. 12. de. 2:51 ezt írta ("Scot" <scotrn@gmail.com>):
I'm almost there I think! An idea why is outputting message ={} nothing ? What is rule='#anon-parser0
*relative lines from syslog-ng.conf* source s_BEATS {network(port(*5140*) log-msg-size(65536) flags(no-parse));}; destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s_BEATS); parser {json-parser();}; destination (d_jfile); };
*/etc/logstash/conf.d/logstash.conf* input { beats { port => 5044 } }
output { tcp { host => "192.168.1.16" port => "*5140*" mode => "client" }
*Running syslog-ng in foreground. *
[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt" :{"us":2000},"monitor":"http@http://192.168.1.16:9200"," type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us": 5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us": 5000},"port":9200,"response":{"status":200},"beat":{" hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version" :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up": true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt": {"us":2000},"monitor":"http@http://192.168.1.16:9200","type" :"http","http_rtt":{"us":5000},"url":"http://192.168.1.16:9200 ","tags":["beats_input_raw_event"],"duration":{"us": 7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:4 2.020Z","port":9200,"response":{"status":200},"beat":{" hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version" :"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up": true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt": {"us":2000},"monitor":"http@http://192.168.1.16:9200","type":"'
[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33' [2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)' [2017-05-11T20:38:49.900384] Outgoing message; message='{}
On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca> wrote:
Since you already have the no-parse flag on the source, everything goes into $MESSAGE
Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.
Evan.
On 05/10/2017 12:57 PM, Scot wrote:
Thanks Evan,
Bumped it up to 32768
Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A I think there may be something else I need to do with the payload.
How would I dump everything to a file to look at it ?
On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:
looks like you might be running into the maximum message size. Try setting the syslog-ng configuration item
log_msg_size(64K);
On 05/10/2017 10:50 AM, Scot wrote:
Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. Trying different options but I'm fumbling.
syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
source s_BEATS {network(port(5140) flags(no-parse));} parser p_json { json-parser (prefix(".json.")); }; log { source(s_BEATS); parser(p_json); destination (d_file); };
Anyone have a howto or blog for using syslog-ng with json inputs ? I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
Trying to get here winlogbeat->syslog-ng->ES winlogbeat->syslog-ng->SPLUNKForwader winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
or winlogbeat->logstash->syslog-ng->ES ...
On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
I'm trying to find a solution that will let me mirror my beats data like syslog-ng lets me do with syslog traffic.
As far as I know those tools simply send the data over TCP in JSON format. If you just need to do routing using syslog-ng, you can simply use network source with flags(no-parse). If you need to process the data using syslog-ng, you'll also need the json-parser().
Cheers
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, May 12, 2017 at 12:50:16AM -0400, Scot wrote:
destination d_es_beats { elasticsearch2( disk-buffer( [...] index("winlogbeat-${YEAR}.${MONTH}.${DAY}")
just a sidenote here: don't forget to add time-zone(UTC) to your elasticsearch destination, otherwise you'll have surprises in Kibana
I don't get it, I don't have that in my current ES target for syslog. destination d_es { elasticsearch2( disk-buffer( reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option dir("/opt/syslog-ng/buffer") disk-buf-size(10485760) mem-buf-length(100000) # number of messages stored in overflow queue ) client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") cluster-url("http://192.168.1.16:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("hal") client_lib_dir("/usr/share/elasticsearch/lib") ); }; On Fri, May 12, 2017 at 4:32 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
On Fri, May 12, 2017 at 12:50:16AM -0400, Scot wrote:
destination d_es_beats { elasticsearch2( disk-buffer( [...] index("winlogbeat-${YEAR}.${MONTH}.${DAY}")
just a sidenote here: don't forget to add time-zone(UTC) to your elasticsearch destination, otherwise you'll have surprises in Kibana
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, On Fri, May 12, 2017 at 06:38:46AM -0400, Scot wrote:
I don't get it, I don't have that in my current ES target for syslog.
Kibana and most other frontends and maybe even libraries use the query's time to infer the index name: syslog-ng_${YEAR}.${MONTH}.${DAY} in your example. They use UTC timezone to do that. If you use localtime, like in your example (implicit) documents having 00:30 as timestamp for example will end up in the wrong index, and this will also depend on DST. When you search for them in kibana you might end up with wrong results. TL;DR: use time-zone("UTC") in your elasticsearch destination whenever you use time-based indices. Maybe we should even add that to the default SCL.
I'm still not getting it. My standard syslog data works fine is there something behind the scenes ? My Syslog ES target wis working but getting nothing in the beats target no errors either. destination d_es { elasticsearch2( disk-buffer( reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option dir("/opt/syslog-ng/buffer") disk-buf-size(10485760) mem-buf-length(100000) # number of messages stored in overflow queue ) client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") cluster-url("http://192.168.1.16:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("hal") client_lib_dir("/usr/share/elasticsearch/lib") ); }; destination d_es_beats { elasticsearch2( disk-buffer( reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option dir("/opt/syslog-ng/buffer/beats") disk-buf-size(10485760) mem-buf-length(100000) # number of messages stored in overflow queue ) # END DiskBuffer client-mode("http") index("winlogbeat-${YEAR}.${MONTH}.${DAY}") type("winlogbeat") # Description: The type of the index. For example, type("test") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") #template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)") template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)") #template("${MESSAGE}") cluster-url("http://192.168.1.16:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("hal") client_lib_dir("/usr/share/elasticsearch/lib") ); }; On Fri, May 12, 2017 at 7:05 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Fri, May 12, 2017 at 06:38:46AM -0400, Scot wrote:
I don't get it, I don't have that in my current ES target for syslog.
Kibana and most other frontends and maybe even libraries use the query's time to infer the index name: syslog-ng_${YEAR}.${MONTH}.${DAY} in your example. They use UTC timezone to do that. If you use localtime, like in your example (implicit) documents having 00:30 as timestamp for example will end up in the wrong index, and this will also depend on DST.
When you search for them in kibana you might end up with wrong results.
TL;DR: use time-zone("UTC") in your elasticsearch destination whenever you use time-based indices.
Maybe we should even add that to the default SCL.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I take that back. Getting data in the winlogbeat target but the json is not getting parsed. All coming in as a messages block. { "_index": "winlogbeat-2017.05.12", "_type": "winlogbeat", "_id": "AVv9Xfil6uwlymto3Hmd", "_score": null, "_source": { "SOURCE": "s_BEATS", "MESSAGE": "{\"scheme\":\"http\",\"ip\":\"192.168.1.16\",\"tcp_connect_rtt\":{\"us\":2000},\"monitor\":\"http@ http://192.168.1.16:9200 \",\"type\":\"http\",\"http_rtt\":{\"us\":2000},\"url\":\" http://192.168.1.16:9200 \",\"tags\":[\"beats_input_raw_event\"],\"duration\":{\"us\":4000},\"@timestamp\":\"2017-05-12T15:54:07.258Z\",\"rtt\":{\"us\":4000},\"port\":9200,\"response\":{\"status\":200},\"beat\":{\"hostname\":\"TYLER-LAPTOP\",\"name\":\"TYLER-LAPTOP\",\"version\":\"5.4.0\"},\"@version\":\"1\",\"host\":\"TYLER-LAPTOP\",\"up\":true}", "HOST_FROM": "hal", "HOST": "hal", "@timestamp": "2017-05-12T11:54:03-04:00", "@message": "{\"scheme\":\"http\",\"ip\":\"192.168.1.16\",\"tcp_connect_rtt\":{\"us\":2000},\"monitor\":\"http@ http://192.168.1.16:9200 \",\"type\":\"http\",\"http_rtt\":{\"us\":2000},\"url\":\" http://192.168.1.16:9200 \",\"tags\":[\"beats_input_raw_event\"],\"duration\":{\"us\":4000},\"@timestamp\":\"2017-05-12T15:54:07.258Z\",\"rtt\":{\"us\":4000},\"port\":9200,\"response\":{\"status\":200},\"beat\":{\"hostname\":\"TYLER-LAPTOP\",\"name\":\"TYLER-LAPTOP\",\"version\":\"5.4.0\"},\"@version\":\"1\",\"host\":\"TYLER-LAPTOP\",\"up\":true}" }, "fields": { "@timestamp": [ 1494604443000 ] }, "sort": [ 1494604443000 ] } On Fri, May 12, 2017 at 11:47 AM, Scot <scotrn@gmail.com> wrote:
I'm still not getting it. My standard syslog data works fine is there something behind the scenes ? My Syslog ES target wis working but getting nothing in the beats target no errors either.
destination d_es { elasticsearch2( disk-buffer( reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option dir("/opt/syslog-ng/buffer") disk-buf-size(10485760) mem-buf-length(100000) # number of messages stored in overflow queue ) client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") cluster-url("http://192.168.1.16:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("hal") client_lib_dir("/usr/share/elasticsearch/lib") ); };
destination d_es_beats { elasticsearch2( disk-buffer( reliable(no) # If set to no, the normal disk-buffer will be used. This provides a faster, option dir("/opt/syslog-ng/buffer/beats") disk-buf-size(10485760) mem-buf-length(100000) # number of messages stored in overflow queue ) # END DiskBuffer client-mode("http") index("winlogbeat-${YEAR}.${MONTH}.${DAY}") type("winlogbeat") # Description: The type of the index. For example, type("test") #template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") #template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)") template("$(format-json -s all-nv-pairs -p @timestamp=$ISODATE -p @message=$MESSAGE)") #template("${MESSAGE}") cluster-url("http://192.168.1.16:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("hal") client_lib_dir("/usr/share/elasticsearch/lib") ); };
On Fri, May 12, 2017 at 7:05 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi,
On Fri, May 12, 2017 at 06:38:46AM -0400, Scot wrote:
I don't get it, I don't have that in my current ES target for syslog.
Kibana and most other frontends and maybe even libraries use the query's time to infer the index name: syslog-ng_${YEAR}.${MONTH}.${DAY} in your example. They use UTC timezone to do that. If you use localtime, like in your example (implicit) documents having 00:30 as timestamp for example will end up in the wrong index, and this will also depend on DST.
When you search for them in kibana you might end up with wrong results.
TL;DR: use time-zone("UTC") in your elasticsearch destination whenever you use time-based indices.
Maybe we should even add that to the default SCL.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, May 12, 2017 at 11:55:51AM -0400, Scot wrote:
I take that back. Getting data in the winlogbeat target but the json is not getting parsed. All coming in as a messages block.
At this point, providing the full config would help. As for my point about UTC, just trust me, you want that.
participants (4)
-
Balazs Scheidler
-
Evan Rempel
-
Fabien Wernli
-
Scot