[syslog-ng] Syslog-ng input for beats ?

[Scot]

Thanks Evan,

Bumped it up to 32768

 Error extracting JSON members into LogMessage as the top-level JSON object
is not an object; input='":"A
I think there may be something else I need to do with the payload.

How would I dump everything to a file to look at it ?




On Wed, May 10, 2017 at 2:10 PM, Evan Rempel <erempel at uvic.ca> wrote:

> looks like you might be running into the maximum message size.
> Try setting the syslog-ng configuration item
>
> log_msg_size(64K);
>
>
>
> On 05/10/2017 10:50 AM, Scot wrote:
>
> Using a RAW TCP seems to be loosing some of the beats header data and
> messages are getting concatenated.
> Trying different options but I'm fumbling.
>
>   syslog-ng[4596]: Unparsable JSON stream encountered;
> input='=net"},"message":"Synchronization of a replica of an Active
> Directory naming context has begun.\n\nDestination DRA:\tCN=NTDS
> Settings,CN=...blaaa"
>
>
> source s_BEATS          {network(port(5140) flags(no-parse));}
> parser p_json {
>     json-parser (prefix(".json."));
> };
> log { source(s_BEATS);  parser(p_json); destination (d_file); };
>
>
> Anyone have a howto or blog for using syslog-ng with json inputs ?
> I'm looking at the syslog-ng-ose-latest-guides but it's hard to put all
> the input output and parser requirements together.
>
> Trying to get here
> winlogbeat->syslog-ng->ES
> winlogbeat->syslog-ng->SPLUNKForwader
> winlogbeat->syslog-ng->/opt/syslog-ng/logs/$FROM_HOST.json
>
> or
> winlogbeat->logstash->syslog-ng->ES
> ...
>
> On Tue, May 9, 2017 at 3:27 AM, Fabien Wernli <wernli at in2p3.fr> wrote:
>
>> Hi,
>>
>> On Mon, May 08, 2017 at 11:30:14PM +0000, Scot wrote:
>> > I'm trying to find a solution that will let me mirror my beats data like
>> > syslog-ng lets me do with syslog traffic.
>>
>> As far as I know those tools simply send the data over TCP in JSON format.
>> If you just need to do routing using syslog-ng, you can simply use network
>> source with flags(no-parse). If you need to process the data using
>> syslog-ng, you'll also need the json-parser().
>>
>> Cheers
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support
>> /documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
> --
> Evan Rempel                                      erempel at uvic.ca
> Senior Systems Administrator                        250.721.7691 <(250)%20721-7691>
> Data Centre Services, University Systems, University of Victoria
>
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170510/ba9c3cdf/attachment-0001.html>