[syslog-ng] ELK herd to scale

[Scot Needy]

Some thoughts on my deployment

Logstash
I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. 
VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb. 
 
syslog-ng counters 
We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats.. 


@include IPAM-filters
filter f_192_168_252_0 { netmask(192.168.252.0/24);};
filter f_192_168_253_0 { netmask(192.168.253.0/24);};
filter f_192_168_254_0 { netmask(192.168.254.0/30);};


@include IPAM-dest.conf
destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};

@include IPAM-log.conf
log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);};
log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);};
log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);};
log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};



> On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn at gmail.com> wrote:
> 
> 
> 
> Hi,   
> 
>  Does anyone have links or care to share notes on making a syslog-ng -> ELK  scale for enterprise ? 
> 
> I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. 
> There are many ELK specific references but I also want to make sure the model fits the syslog workload. 
> 
> 
> Thanks 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160420/aaf81fe9/attachment.htm