[syslog-ng] syslog-ng as "shipper" into ELK stack

[Jim Hendrick]

Hi,

  Just realized I never replied to you.

I spent some time this past week trying to get the incubator module
working with very limited success
(mostly due to this being a part of one of my job responsibilities)

I did get syslog-ng & patterndb doing what I was doing in logstash &
grok and it seems much faster.

I will work on this more hopefully later this week, but I did want to reply

Thanks!
Jim


On 10/03/2014 04:12 AM, Fabien Wernli wrote:
> Hi Jim,
>
> On Fri, Oct 03, 2014 at 12:33:41AM +0000, Jim Hendrick wrote:
>> syslog-ng ==> redis ==> logstash ==> elasticsearch ==> apache ==> kibana
> We've been using the following stack for over a year:
> syslog-ng ==> logstash ==> elasticsearch
>
> For various reasons, one being performance, we recently switched to:
> syslog-ng ==> elasticsearch
>
> This was done thanks to the syslog-ng-incubator perl module. I've set up a
> small github repository where you can see our configuration [1].
>
>> (I topped out today sending ~7000 events per second, and saw an insane 
>> amount of swapping going on)
> I've had tremendous issues with LS when the workload was darting up.
> Since we switched to perl, we still have issues, but they're certainly not
> performance related: with a single perl destination we could easily keep up
> 10k events per second on a mediumish virtual machine.
>
>> Is anyone aware of any plans to implement an elasticsearch destination?
> The upcoming 3.6 version will ship with a "native" elasticsearch
> destination, which currently however is only a wrapper script.
>
> I'd highly appreciate if you could test a similar config to ours, in order
> to share some experience.
>
> Cheers
>
> [1] https://github.com/faxm0dem/syslog_ng-elasticsearch
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>