[syslog-ng] ELK herd to scale

[Orangepeel Beef]

logstash-* index is for logs that have been ingested via logstash of course
:)

every component of ELK scales horizontally extremely well.

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon>
Virus-free.
www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
<#DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn at gmail.com> wrote:

>
>
> https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template
>
> May have misspoke. Using ELK and patterndb.xml  is new to me and I am
> still trying to learn the mechanics.
>
>
>  I started by looking at Google for Kibana dashboard templates, one of the
> better results here.
> https://github.com/markwalkom/kibana-dashboards  Most of the kibana json
> templates I have seen on the net are setup for a logstash-*  “index” ?.
>
> I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time
> and ease of setup and support community size are big considerations. I want
> to enable GeoIP for ASA data, NetFlow data and be able to leverage existing
> templates logstash or patterndb for common applications.  Apache, Linux
> Syslog, Storage syslog, etc…
>
>
>
> On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <
> balazs.scheidler at balabit.com> wrote:
>
> Can you pls point me to the direction of the logstash material you
> mentioned? I would be interested in them whether it'd be possible to port
> them over.
> On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn at gmail.com> wrote:
>
>> Some thoughts on my deployment
>>
>> *Logstash*
>> I think I’m going to need to re-introduce logstash just to leverage the
>> existing open source material of logstash filters and Kibana desktops.
>> VMware, ASA for example but wanted more real time data. I could probably
>> do the realtime tags with pattendb.
>>
>> *syslog-ng counters*
>> We use an IPAM API to create unique filters, log and destination conf
>> files. The goal was to get unique syslog counters for every VLAN realtime
>> directly from syslog-ng-ctl stats..
>>
>>
>> @include IPAM-filters
>> filter f_192_168_252_0 { netmask(192.168.252.0/24);};
>> filter f_192_168_253_0 { netmask(192.168.253.0/24);};
>> filter f_192_168_254_0 { netmask(192.168.254.0/30);};
>>
>>
>> @include IPAM-dest.conf
>> destination d_192_168_252_0 {
>> file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>> destination d_192_168_253_0 {
>> file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>> destination d_192_168_254_0 {
>> file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>
>> @include IPAM-log.conf
>> log { source(s_net); filter(f_192_168_252_0);
>> destination(d_192_168_252_0);};
>> log { source(s_net); filter(f_192_168_253_0);
>> destination(d_192_168_253_0);};
>> log { source(s_net); filter(f_192_168_254_0);
>> destination(d_192_168_254_0);};
>> log { source(s_net); filter(f_192_168_254_4);
>> destination(d_192_168_254_4);};
>>
>>
>>
>> On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn at gmail.com> wrote:
>>
>>
>>
>> Hi,
>>
>>  Does anyone have links or care to share notes on making a syslog-ng ->
>> ELK  scale for enterprise ?
>>
>> I have some ideas and will gladly share my solution but also don’t want
>> to spend days figuring these things out that have already been built.
>> There are many ELK specific references but I also want to make sure the
>> model fits the syslog workload.
>>
>>
>> Thanks
>>
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160420/252c919e/attachment.htm