[syslog-ng] ELK herd to scale

Thu Apr 21 06:02:33 CEST 2016

We used rsyslog to receive from all networking devices, and
rsyslog<->rsyslog for systems.  then I did some addtional processing of the
logs for realtime alerting and shove them into ES with logstash.  They all
complement each other.  Then you just need to stand up 1 log collection (in
your case syslog-ng) server in each physical datacenter, and use logstash
to parse the logs, and lumberjack to send them out to some central ES stack.

On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn at gmail.com> wrote:

>
> That seems to be the way most people are doing it but I think in absence
> of syslog-ng not in place of it.
> But I don’t like the idea of waiting every XX minutes to see my ASA logs.
>
> My understanding is the Logstash part of the ELK stack is not required if
> you use the syslog-ng Elasticsearch plugin.
> pro Realtime data
> pro No additional hop for your data.
>
>
> On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef at gmail.com>
> wrote:
>
> They way I always liked doing it was to send all the logs via syslog
> regularly to your central collection server and use logstash file input to
> parse them in and shove into ES.
>
> On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <orangepeelbeef at gmail.com
> > wrote:
>
>> logstash-* index is for logs that have been ingested via logstash of
>> course :)
>>
>> every component of ELK scales horizontally extremely well.
>>
>>
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
>> www.avast.com
>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>>
>> On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn at gmail.com> wrote:
>>
>>>
>>>
>>> https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template
>>>
>>> May have misspoke. Using ELK and patterndb.xml  is new to me and I am
>>> still trying to learn the mechanics.
>>>
>>>
>>>  I started by looking at Google for Kibana dashboard templates, one of
>>> the better results here.
>>> https://github.com/markwalkom/kibana-dashboards  Most of the kibana
>>> json templates I have seen on the net are setup for a logstash-*  “index”
>>> ?.
>>>
>>> I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time
>>> and ease of setup and support community size are big considerations. I want
>>> to enable GeoIP for ASA data, NetFlow data and be able to leverage existing
>>> templates logstash or patterndb for common applications.  Apache, Linux
>>> Syslog, Storage syslog, etc…
>>>
>>>
>>>
>>> On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <
>>> balazs.scheidler at balabit.com> wrote:
>>>
>>> Can you pls point me to the direction of the logstash material you
>>> mentioned? I would be interested in them whether it'd be possible to port
>>> them over.
>>> On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn at gmail.com> wrote:
>>>
>>>> Some thoughts on my deployment
>>>>
>>>> *Logstash*
>>>> I think I’m going to need to re-introduce logstash just to leverage the
>>>> existing open source material of logstash filters and Kibana desktops.
>>>> VMware, ASA for example but wanted more real time data. I could
>>>> probably do the realtime tags with pattendb.
>>>>
>>>> *syslog-ng counters*
>>>> We use an IPAM API to create unique filters, log and destination conf
>>>> files. The goal was to get unique syslog counters for every VLAN realtime
>>>> directly from syslog-ng-ctl stats..
>>>>
>>>>
>>>> @include IPAM-filters
>>>> filter f_192_168_252_0 { netmask(192.168.252.0/24);};
>>>> filter f_192_168_253_0 { netmask(192.168.253.0/24);};
>>>> filter f_192_168_254_0 { netmask(192.168.254.0/30);};
>>>>
>>>>
>>>> @include IPAM-dest.conf
>>>> destination d_192_168_252_0 {
>>>> file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>> destination d_192_168_253_0 {
>>>> file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>> destination d_192_168_254_0 {
>>>> file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>
>>>> @include IPAM-log.conf
>>>> log { source(s_net); filter(f_192_168_252_0);
>>>> destination(d_192_168_252_0);};
>>>> log { source(s_net); filter(f_192_168_253_0);
>>>> destination(d_192_168_253_0);};
>>>> log { source(s_net); filter(f_192_168_254_0);
>>>> destination(d_192_168_254_0);};
>>>> log { source(s_net); filter(f_192_168_254_4);
>>>> destination(d_192_168_254_4);};
>>>>
>>>>
>>>>
>>>> On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn at gmail.com> wrote:
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>  Does anyone have links or care to share notes on making a syslog-ng ->
>>>> ELK  scale for enterprise ?
>>>>
>>>> I have some ideas and will gladly share my solution but also don’t want
>>>> to spend days figuring these things out that have already been built.
>>>> There are many ELK specific references but I also want to make sure the
>>>> model fits the syslog workload.
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160420/55f60be5/attachment-0001.htm 