[syslog-ng] ELK herd to scale

[Scot Needy]

https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template <https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana dashboard template>

May have misspoke. Using ELK and patterndb.xml  is new to me and I am still trying to learn the mechanics. 


 I started by looking at Google for Kibana dashboard templates, one of the better results here. 
https://github.com/markwalkom/kibana-dashboards <https://github.com/markwalkom/kibana-dashboards>  Most of the kibana json templates I have seen on the net are setup for a logstash-*  “index” ?.  

I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications.  Apache, Linux Syslog, Storage syslog, etc… 



> On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <balazs.scheidler at balabit.com> wrote:
> 
> Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over.
> 
> On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn at gmail.com <mailto:scotrn at gmail.com>> wrote:
> Some thoughts on my deployment
> 
> Logstash
> I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. 
> VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb. 
>  
> syslog-ng counters 
> We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats.. 
> 
> 
> @include IPAM-filters
> filter f_192_168_252_0 { netmask(192.168.252.0/24); <http://192.168.252.0/24);>};
> filter f_192_168_253_0 { netmask(192.168.253.0/24); <http://192.168.253.0/24);>};
> filter f_192_168_254_0 { netmask(192.168.254.0/30); <http://192.168.254.0/30);>};
> 
> 
> @include IPAM-dest.conf
> destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
> destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
> destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
> 
> @include IPAM-log.conf
> log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);};
> log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);};
> log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);};
> log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
> 
> 
> 
>> On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn at gmail.com <mailto:scotrn at gmail.com>> wrote:
>> 
>> 
>> 
>> Hi,   
>> 
>>  Does anyone have links or care to share notes on making a syslog-ng -> ELK  scale for enterprise ? 
>> 
>> I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. 
>> There are many ELK specific references but I also want to make sure the model fits the syslog workload. 
>> 
>> 
>> Thanks 
>> 
> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng>
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng>
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160420/7cd3ff8d/attachment-0001.htm