[syslog-ng] Syslog-ng input for beats ?

[Evan Rempel]

Since you already have the no-parse flag on the source, everything goes into $MESSAGE

Make a file destination with the template of "$MESSAGE\n" and it should contain the entire payload.

Evan.

On 05/10/2017 12:57 PM, Scot wrote:
> Thanks Evan,
>
> Bumped it up to 32768
>
>  Error extracting JSON members into LogMessage as the top-level JSON object is not an object; input='":"A
> I think there may be something else I need to do with the payload.
>
> How would I dump everything to a file to look at it ?
>
>
>
>
> On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel at uvic.ca <mailto:erempel at uvic.ca>> wrote:
>
>     looks like you might be running into the maximum message size.
>     Try setting the syslog-ng configuration item
>
>     log_msg_size(64K);
>
>
>
>     On 05/10/2017 10:50 AM, Scot wrote:
>>     Using a RAW TCP seems to be loosing some of the beats header data and messages are getting concatenated.
>>     Trying different options but I'm fumbling.
>>
>>       syslog-ng[4596]: Unparsable JSON stream encountered; input='=net"},"message":"Synchronization of a replica of an Active Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS Settings,CN=...blaaa"
>>
>>
>>     source s_BEATS          {network(port(5140) flags(no-parse));}
>>     parser p_json {
>>         json-parser (prefix(".json."));
>>     };
>>     log { source(s_BEATS);  parser(p_json); destination (d_file); };
>>
>>
>>     Anyone have a howto or blog for using syslog-ng with json inputs ?
>>     I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all the input output and parser requirements together.
>>
>>     Trying to get here
>>     winlogbeat->syslog-ng->ES
>>     winlogbeat->syslog-ng->SPLUNKForwader
>>     winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
>>
>>     or
>>     winlogbeat->logstash->syslog-ng->ES
>>     ...
>>
>>     On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli at in2p3.fr <mailto:wernli at in2p3.fr>> wrote:
>>
>>         Hi,
>>
>>         On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
>>         > I'm trying to find a solution that will let me mirror my beats data like
>>         > syslog-ng lets me do with syslog traffic.
>>
>>         As far as I know those tools simply send the data over TCP in JSON format.
>>         If you just need to do routing using syslog-ng, you can simply use network
>>         source with flags(no-parse). If you need to process the data using
>>         syslog-ng, you'll also need the json-parser().
>>
>>         Cheers
>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170510/1934fa60/attachment.html>