[syslog-ng] ELK herd to scale

[Orangepeel Beef]

They way I always liked doing it was to send all the logs via syslog
regularly to your central collection server and use logstash file input to
parse them in and shove into ES.

On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <orangepeelbeef at gmail.com>
wrote:

> logstash-* index is for logs that have been ingested via logstash of
> course :)
>
> every component of ELK scales horizontally extremely well.
>
>
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
> www.avast.com
> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
> <#m_-2741954605978871063_DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>
> On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn at gmail.com> wrote:
>
>>
>>
>> https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template
>>
>> May have misspoke. Using ELK and patterndb.xml  is new to me and I am
>> still trying to learn the mechanics.
>>
>>
>>  I started by looking at Google for Kibana dashboard templates, one of
>> the better results here.
>> https://github.com/markwalkom/kibana-dashboards  Most of the kibana json
>> templates I have seen on the net are setup for a logstash-*  “index” ?.
>>
>> I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time
>> and ease of setup and support community size are big considerations. I want
>> to enable GeoIP for ASA data, NetFlow data and be able to leverage existing
>> templates logstash or patterndb for common applications.  Apache, Linux
>> Syslog, Storage syslog, etc…
>>
>>
>>
>> On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <
>> balazs.scheidler at balabit.com> wrote:
>>
>> Can you pls point me to the direction of the logstash material you
>> mentioned? I would be interested in them whether it'd be possible to port
>> them over.
>> On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn at gmail.com> wrote:
>>
>>> Some thoughts on my deployment
>>>
>>> *Logstash*
>>> I think I’m going to need to re-introduce logstash just to leverage the
>>> existing open source material of logstash filters and Kibana desktops.
>>> VMware, ASA for example but wanted more real time data. I could probably
>>> do the realtime tags with pattendb.
>>>
>>> *syslog-ng counters*
>>> We use an IPAM API to create unique filters, log and destination conf
>>> files. The goal was to get unique syslog counters for every VLAN realtime
>>> directly from syslog-ng-ctl stats..
>>>
>>>
>>> @include IPAM-filters
>>> filter f_192_168_252_0 { netmask(192.168.252.0/24);};
>>> filter f_192_168_253_0 { netmask(192.168.253.0/24);};
>>> filter f_192_168_254_0 { netmask(192.168.254.0/30);};
>>>
>>>
>>> @include IPAM-dest.conf
>>> destination d_192_168_252_0 {
>>> file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>> destination d_192_168_253_0 {
>>> file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>> destination d_192_168_254_0 {
>>> file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>
>>> @include IPAM-log.conf
>>> log { source(s_net); filter(f_192_168_252_0);
>>> destination(d_192_168_252_0);};
>>> log { source(s_net); filter(f_192_168_253_0);
>>> destination(d_192_168_253_0);};
>>> log { source(s_net); filter(f_192_168_254_0);
>>> destination(d_192_168_254_0);};
>>> log { source(s_net); filter(f_192_168_254_4);
>>> destination(d_192_168_254_4);};
>>>
>>>
>>>
>>> On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn at gmail.com> wrote:
>>>
>>>
>>>
>>> Hi,
>>>
>>>  Does anyone have links or care to share notes on making a syslog-ng ->
>>> ELK  scale for enterprise ?
>>>
>>> I have some ideas and will gladly share my solution but also don’t want
>>> to spend days figuring these things out that have already been built.
>>> There are many ELK specific references but I also want to make sure the
>>> model fits the syslog workload.
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160420/2be874fc/attachment-0001.htm