<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">ES2 and Kibanna 4 but hold on a sec… </div><div class=""><br class=""></div><div class="">I used your blog as a starting reference. Great job BTW, but I don’t see how the logstash index can exist using this directive in syslogng.conf </div><div class=""><br class=""></div><div class=""><span style="color: rgb(59, 59, 59); line-height: 17.4017px; widows: 1; background-color: rgb(255, 255, 255);" class=""> option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}")</span></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On Apr 21, 2016, at 3:41 AM, Czanik, Péter <<a href="mailto:peter.czanik@balabit.com" class="">peter.czanik@balabit.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class=""><div class=""><div class="">Hi,<br class=""></div><br class="">To get started with syslog-ng + patterndb + ElasticSearch, you could use my blog at <a href="https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/" class="">https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/</a> It shows how to parse ssh log-in messages, and how to display them with Kibana.<br class=""><br class=""></div>If you use "logstash-${YEAR}.${MONTH}.${DAY}" as index name, Kibana should find your logs without any extra setup.<br class=""></div><br class="">Which versions of Elasticsearch and Kibana do you use?<br class=""><br class=""></div>Bye,<br class=""></div><div class="gmail_extra"><br clear="all" class=""><div class=""><div class="gmail_signature">Peter Czanik (CzP) <<a href="mailto:peter.czanik@balabit.com" target="_blank" class="">peter.czanik@balabit.com</a>><br class="">Balabit / syslog-ng upstream<br class=""><a href="http://czanik.blogs.balabit.com/" target="_blank" class="">http://czanik.blogs.balabit.com/</a><br class=""><a href="https://twitter.com/PCzanik" target="_blank" class="">https://twitter.com/PCzanik</a></div></div>
<br class=""><div class="gmail_quote">On Thu, Apr 21, 2016 at 9:13 AM, Balazs Scheidler <span dir="ltr" class=""><<a href="mailto:bazsi77@gmail.com" target="_blank" class="">bazsi77@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class=""><div class="">well, one thing that is probably a lot faster with syslog-ng is parsing, so I guess it all boils down to performance needed at the reception side.<br class=""><br class=""></div>receiving and writing out into a file, then polling that file for changes is definitely slower if you have non-trival amount of messages. syslog-ng receve & parse can happen at the 100k/second range and certainly ES is usually slower than that, but that's the entire point in scaling, right? so if I scale ES to the 100k/sec range, doing this with syslog-ng reduces the load a lot at the data center collector a lot.<br class=""></div><div class="gmail_extra"><div class=""><div class="h5"><br class=""><div class="gmail_quote">On Thu, Apr 21, 2016 at 6:02 AM, Orangepeel Beef <span dir="ltr" class=""><<a href="mailto:orangepeelbeef@gmail.com" target="_blank" class="">orangepeelbeef@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">We used rsyslog to receive from all networking devices, and rsyslog<->rsyslog for systems. then I did some addtional processing of the logs for realtime alerting and shove them into ES with logstash. They all complement each other. Then you just need to stand up 1 log collection (in your case syslog-ng) server in each physical datacenter, and use logstash to parse the logs, and lumberjack to send them out to some central ES stack.</div><div class=""><div class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <span dir="ltr" class=""><<a href="mailto:scotrn@gmail.com" target="_blank" class="">scotrn@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><br class=""></div><div class="">That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. </div><div class="">But I don’t like the idea of waiting every XX minutes to see my ASA logs. </div><div class=""><br class=""></div><div class="">My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. </div><div class="">pro Realtime data </div><div class="">pro No additional hop for your data. </div><div class=""><div class=""><div class=""><br class=""></div><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <<a href="mailto:orangepeelbeef@gmail.com" target="_blank" class="">orangepeelbeef@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.</div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <span dir="ltr" class=""><<a href="mailto:orangepeelbeef@gmail.com" target="_blank" class="">orangepeelbeef@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">logstash-* index is for logs that have been ingested via logstash of course :) <div class=""><br class=""></div><div class="">every component of ELK scales horizontally extremely well. </div></div><div class=""><br class="">
<table style="border-top:1px solid #aaabb6" class="">
<tbody class=""><tr class="">
<td style="width:55px;padding-top:13px" class=""><a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon" target="_blank" class=""><img class=""></a></td>
<td style="width:470px;padding-top:15px;color:#41424e;font-size:13px;font-family:Arial,Helvetica,sans-serif;line-height:18px" class="">Virus-free. <a href="https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link" style="color:#4453ea" target="_blank" class="">www.avast.com</a>
</td>
</tr>
</tbody></table><a width="1" height="1" class=""></a></div><div class=""><div class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <span dir="ltr" class=""><<a href="mailto:scotrn@gmail.com" target="_blank" class="">scotrn@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><br class=""></div><div class=""><div class=""><a href="https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template" target="_blank" class="">https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template</a></div></div><div class=""><br class=""></div><div class="">May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics. </div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""> I started by looking at Google for Kibana dashboard templates, one of the better results here. </div><div class=""><a href="https://github.com/markwalkom/kibana-dashboards" target="_blank" class="">https://github.com/markwalkom/kibana-dashboards</a> Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?. </div><div class=""><br class=""></div><div class="">I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc… </div><div class=""><div class=""><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <<a href="mailto:balazs.scheidler@balabit.com" target="_blank" class="">balazs.scheidler@balabit.com</a>> wrote:</div><br class=""><div class=""><p dir="ltr" class="">Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over.</p>
<div class="gmail_quote">On Apr 20, 2016 7:00 PM, "Scot Needy" <<a href="mailto:scotrn@gmail.com" target="_blank" class="">scotrn@gmail.com</a>> wrote:<br type="attribution" class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Some thoughts on my deployment</div><div class=""><br class=""></div><div class=""><b class="">Logstash</b></div><div class="">I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. </div><div class="">VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb. </div><div class=""> </div><div class=""><b class="">syslog-ng counters</b> </div><div class="">We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats.. </div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">@include IPAM-filters</div><div class=""><div class="">filter f_192_168_252_0 { netmask(<a href="http://192.168.252.0/24);" target="_blank" class="">192.168.252.0/24);</a>};</div><div class="">filter f_192_168_253_0 { netmask(<a href="http://192.168.253.0/24);" target="_blank" class="">192.168.253.0/24);</a>};</div><div class="">filter f_192_168_254_0 { netmask(<a href="http://192.168.254.0/30);" target="_blank" class="">192.168.254.0/30);</a>};</div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">@include IPAM-dest.conf</div><div class=""><div class="">destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};</div><div class="">destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};</div><div class="">destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};</div></div><div class=""><br class=""></div><div class="">@include IPAM-log.conf</div><div class=""><div class="">log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);};</div><div class="">log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);};</div><div class="">log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);};</div><div class="">log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};</div></div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div class=""><blockquote type="cite" class=""><div class="">On Apr 20, 2016, at 11:18 AM, Scot Needy <<a href="mailto:scotrn@gmail.com" target="_blank" class="">scotrn@gmail.com</a>> wrote:</div><br class=""><div class=""><br class=""><br class="">Hi, <br class=""><br class=""> Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ? <br class=""><br class="">I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. <br class="">There are many ELK specific references but I also want to make sure the model fits the syslog workload. <br class=""><br class=""><br class="">Thanks <br class=""><br class=""></div></blockquote></div><br class=""></div><br class="">______________________________________________________________________________<br class="">
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
<br class=""></blockquote></div>
______________________________________________________________________________<br class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br class=""></div></blockquote></div><br class=""></div></div></div><br class="">______________________________________________________________________________<br class="">
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
<br class=""></blockquote></div><br class=""></div>
</div></div></blockquote></div><br class=""></div>
______________________________________________________________________________<br class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br class=""></div></blockquote></div><br class=""></div></div></div><br class="">______________________________________________________________________________<br class="">
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
<br class=""></blockquote></div><br class=""></div>
</div></div><br class="">______________________________________________________________________________<br class="">
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
<br class=""></blockquote></div><br class=""><br clear="all" class=""><br class=""></div></div><span class="HOEnZb"><font color="#888888" class="">-- <br class=""><div class="">Bazsi</div>
</font></span></div>
<br class="">______________________________________________________________________________<br class="">
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class="">
<br class="">
<br class=""></blockquote></div><br class=""></div>
______________________________________________________________________________<br class="">Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" class="">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br class="">Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" class="">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br class="">FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" class="">http://www.balabit.com/wiki/syslog-ng-faq</a><br class=""><br class=""></div></blockquote></div><br class=""></body></html>