Hi,

YouTube specified --scope dot-nv-pairs tó format-json, which only includes name-value pairs that start with a  dot.

2017. máj. 12. de. 2:51 ezt írta ("Scot" <scotrn@gmail.com>):

I'm almost there I think! 

An idea why is outputting message ={} nothing ?  What is rule='#anon-parser0

relative lines from syslog-ng.conf

source s_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));};

destination d_jfile { file("/opt/syslog-ng/logs/$HOST_FROM-$R_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));};

log { source(s_BEATS);  parser {json-parser();}; destination (d_jfile); };

/etc/logstash/conf.d/logstash.conf

input {

  beats {

    port => 5044

  }

}

output {

  tcp {

    host => "192.168.1.16"

    port => "5140"

    mode => "client"

  }

Running syslog-ng in foreground.  

[2017-05-11T20:30:53.529215] Syslog connection accepted; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)'

[2017-05-11T20:38:49.899997] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@http://192.168.1.16:9200","type":"http","http_rtt":{"us":2000},"url":"http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{"us":5000},"@timestamp":"2017-05-12T00:30:32.020Z","rtt":{"us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@http://192.168.1.16:9200","type":"http","http_rtt":{"us":5000},"url":"http://192.168.1.16:9200","tags":["beats_input_raw_event"],"duration":{"us":7000},"rtt":{"us":7000},"@timestamp":"2017-05-12T00:30:42.020Z","port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}{"scheme":"http","ip":"192.168.1.16","tcp_connect_rtt":{"us":2000},"monitor":"http@http://192.168.1.16:9200","type":"'

[2017-05-11T20:38:49.900179] Message parsing complete; result='1', rule='#anon-parser0', location='/etc/syslog-ng/syslog-ng.conf:60:33'

[2017-05-11T20:38:49.900324] Syslog connection closed; fd='88', client='AF_INET(192.168.1.16:60660)', local='AF_INET(0.0.0.0:5140)'

[2017-05-11T20:38:49.900384] Outgoing message; message='{}

On Wed, May 10, 2017 at 4:01 PM, Evan Rempel <erempel@uvic.ca> wrote:

Since you already have the no-parse flag on the source, everything goes into $MESSAGE

Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.

Evan.

On 05/10/2017 12:57 PM, Scot wrote:

Thanks Evan, 

Bumped it up to 32768  

 Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A  

I think there may be something else I need to do with the payload. 

How would I dump everything to a file to look at it ? 

On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel@uvic.ca> wrote:

looks like you might be running into the maximum message size.
Try setting the syslog-ng configuration item

log_msg_size(64K);

On 05/10/2017 10:50 AM, Scot wrote:

Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated. 

Trying different options but I'm fumbling. 

  syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"

source s_BEATS          {network(port(5140) flags(no-parse));}

parser p_json {

    json-parser (prefix(".json."));

};

log { source(s_BEATS);  parser(p_json); destination (d_file); };

Anyone have a howto or blog for using syslog-ng with json inputs ?  

I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.  

Trying to get here 

winlogbeat->syslog-ng->ES   

winlogbeat->syslog-ng->SPLUNKForwader

winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json 

or 

winlogbeat->logstash->syslog-ng->ES   

...

On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli@in2p3.fr> wrote:

Hi,

On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
> I'm trying to find a solution that will let me mirror my beats data like
> syslog-ng lets me do with syslog traffic.

As far as I know those tools simply send the data over TCP in JSON format.
If you just need to do routing using syslog-ng, you can simply use network
source with flags(no-parse). If you need to process the data using
syslog-ng, you'll also need the json-parser().

Cheers

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq