[syslog-ng] ELK herd to scale

Thu Apr 21 06:21:33 CEST 2016

Ingestion usually takes maybe a few seconds, so you're in near realtime in
ELK.  You can scale the log ingester by having multiple syslog-ng and
logstash/lumberjack endpoints in each location once the processing starts
to lag.  Either using a LB or splitting your env.

On Wed, Apr 20, 2016 at 9:02 PM, Orangepeel Beef <orangepeelbeef at gmail.com>
wrote:

> We used rsyslog to receive from all networking devices, and
> rsyslog<->rsyslog for systems.  then I did some addtional processing of the
> logs for realtime alerting and shove them into ES with logstash.  They all
> complement each other.  Then you just need to stand up 1 log collection (in
> your case syslog-ng) server in each physical datacenter, and use logstash
> to parse the logs, and lumberjack to send them out to some central ES stack.
>
> On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn at gmail.com> wrote:
>
>>
>> That seems to be the way most people are doing it but I think in absence
>> of syslog-ng not in place of it.
>> But I don’t like the idea of waiting every XX minutes to see my ASA logs.
>>
>> My understanding is the Logstash part of the ELK stack is not required if
>> you use the syslog-ng Elasticsearch plugin.
>> pro Realtime data
>> pro No additional hop for your data.
>>
>>
>> On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef at gmail.com>
>> wrote:
>>
>> They way I always liked doing it was to send all the logs via syslog
>> regularly to your central collection server and use logstash file input to
>> parse them in and shove into ES.
>>
>> On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <
>> orangepeelbeef at gmail.com> wrote:
>>
>>> logstash-* index is for logs that have been ingested via logstash of
>>> course :)
>>>
>>> every component of ELK scales horizontally extremely well.
>>>
>>>
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
>>> www.avast.com
>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>>>
>>> On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn at gmail.com> wrote:
>>>
>>>>
>>>>
>>>> https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template
>>>>
>>>> May have misspoke. Using ELK and patterndb.xml  is new to me and I am
>>>> still trying to learn the mechanics.
>>>>
>>>>
>>>>  I started by looking at Google for Kibana dashboard templates, one of
>>>> the better results here.
>>>> https://github.com/markwalkom/kibana-dashboards  Most of the kibana
>>>> json templates I have seen on the net are setup for a logstash-*  “index”
>>>> ?.
>>>>
>>>> I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So
>>>> time and ease of setup and support community size are big considerations. I
>>>> want to enable GeoIP for ASA data, NetFlow data and be able to leverage
>>>> existing templates logstash or patterndb for common applications.  Apache,
>>>> Linux Syslog, Storage syslog, etc…
>>>>
>>>>
>>>>
>>>> On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <
>>>> balazs.scheidler at balabit.com> wrote:
>>>>
>>>> Can you pls point me to the direction of the logstash material you
>>>> mentioned? I would be interested in them whether it'd be possible to port
>>>> them over.
>>>> On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn at gmail.com> wrote:
>>>>
>>>>> Some thoughts on my deployment
>>>>>
>>>>> *Logstash*
>>>>> I think I’m going to need to re-introduce logstash just to leverage
>>>>> the existing open source material of logstash filters and Kibana desktops.
>>>>> VMware, ASA for example but wanted more real time data. I could
>>>>> probably do the realtime tags with pattendb.
>>>>>
>>>>> *syslog-ng counters*
>>>>> We use an IPAM API to create unique filters, log and destination conf
>>>>> files. The goal was to get unique syslog counters for every VLAN realtime
>>>>> directly from syslog-ng-ctl stats..
>>>>>
>>>>>
>>>>> @include IPAM-filters
>>>>> filter f_192_168_252_0 { netmask(192.168.252.0/24);};
>>>>> filter f_192_168_253_0 { netmask(192.168.253.0/24);};
>>>>> filter f_192_168_254_0 { netmask(192.168.254.0/30);};
>>>>>
>>>>>
>>>>> @include IPAM-dest.conf
>>>>> destination d_192_168_252_0 {
>>>>> file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>> destination d_192_168_253_0 {
>>>>> file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>> destination d_192_168_254_0 {
>>>>> file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>>
>>>>> @include IPAM-log.conf
>>>>> log { source(s_net); filter(f_192_168_252_0);
>>>>> destination(d_192_168_252_0);};
>>>>> log { source(s_net); filter(f_192_168_253_0);
>>>>> destination(d_192_168_253_0);};
>>>>> log { source(s_net); filter(f_192_168_254_0);
>>>>> destination(d_192_168_254_0);};
>>>>> log { source(s_net); filter(f_192_168_254_4);
>>>>> destination(d_192_168_254_4);};
>>>>>
>>>>>
>>>>>
>>>>> On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn at gmail.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>  Does anyone have links or care to share notes on making a syslog-ng
>>>>> -> ELK  scale for enterprise ?
>>>>>
>>>>> I have some ideas and will gladly share my solution but also don’t
>>>>> want to spend days figuring these things out that have already been built.
>>>>> There are many ELK specific references but I also want to make sure
>>>>> the model fits the syslog workload.
>>>>>
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160420/76fc76f7/attachment.htm 