[syslog-ng] ELK herd to scale
Czanik, Péter
peter.czanik at balabit.com
Thu Apr 21 09:41:15 CEST 2016
Hi,
To get started with syslog-ng + patterndb + ElasticSearch, you could use my
blog at
https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/
It shows how to parse ssh log-in messages, and how to display them with
Kibana.
If you use "logstash-${YEAR}.${MONTH}.${DAY}" as index name, Kibana should
find your logs without any extra setup.
Which versions of Elasticsearch and Kibana do you use?
Bye,
Peter Czanik (CzP) <peter.czanik at balabit.com>
Balabit / syslog-ng upstream
http://czanik.blogs.balabit.com/
https://twitter.com/PCzanik
On Thu, Apr 21, 2016 at 9:13 AM, Balazs Scheidler <bazsi77 at gmail.com> wrote:
> well, one thing that is probably a lot faster with syslog-ng is parsing,
> so I guess it all boils down to performance needed at the reception side.
>
> receiving and writing out into a file, then polling that file for changes
> is definitely slower if you have non-trival amount of messages. syslog-ng
> receve & parse can happen at the 100k/second range and certainly ES is
> usually slower than that, but that's the entire point in scaling, right? so
> if I scale ES to the 100k/sec range, doing this with syslog-ng reduces the
> load a lot at the data center collector a lot.
>
> On Thu, Apr 21, 2016 at 6:02 AM, Orangepeel Beef <orangepeelbeef at gmail.com
> > wrote:
>
>> We used rsyslog to receive from all networking devices, and
>> rsyslog<->rsyslog for systems. then I did some addtional processing of the
>> logs for realtime alerting and shove them into ES with logstash. They all
>> complement each other. Then you just need to stand up 1 log collection (in
>> your case syslog-ng) server in each physical datacenter, and use logstash
>> to parse the logs, and lumberjack to send them out to some central ES stack.
>>
>> On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn at gmail.com> wrote:
>>
>>>
>>> That seems to be the way most people are doing it but I think in absence
>>> of syslog-ng not in place of it.
>>> But I don’t like the idea of waiting every XX minutes to see my ASA
>>> logs.
>>>
>>> My understanding is the Logstash part of the ELK stack is not required
>>> if you use the syslog-ng Elasticsearch plugin.
>>> pro Realtime data
>>> pro No additional hop for your data.
>>>
>>>
>>> On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef at gmail.com>
>>> wrote:
>>>
>>> They way I always liked doing it was to send all the logs via syslog
>>> regularly to your central collection server and use logstash file input to
>>> parse them in and shove into ES.
>>>
>>> On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <
>>> orangepeelbeef at gmail.com> wrote:
>>>
>>>> logstash-* index is for logs that have been ingested via logstash of
>>>> course :)
>>>>
>>>> every component of ELK scales horizontally extremely well.
>>>>
>>>>
>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
>>>> www.avast.com
>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>>>>
>>>> On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn at gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template
>>>>>
>>>>> May have misspoke. Using ELK and patterndb.xml is new to me and I am
>>>>> still trying to learn the mechanics.
>>>>>
>>>>>
>>>>> I started by looking at Google for Kibana dashboard templates, one of
>>>>> the better results here.
>>>>> https://github.com/markwalkom/kibana-dashboards Most of the kibana
>>>>> json templates I have seen on the net are setup for a logstash-* “index”
>>>>> ?.
>>>>>
>>>>> I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So
>>>>> time and ease of setup and support community size are big considerations. I
>>>>> want to enable GeoIP for ASA data, NetFlow data and be able to leverage
>>>>> existing templates logstash or patterndb for common applications. Apache,
>>>>> Linux Syslog, Storage syslog, etc…
>>>>>
>>>>>
>>>>>
>>>>> On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <
>>>>> balazs.scheidler at balabit.com> wrote:
>>>>>
>>>>> Can you pls point me to the direction of the logstash material you
>>>>> mentioned? I would be interested in them whether it'd be possible to port
>>>>> them over.
>>>>> On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn at gmail.com> wrote:
>>>>>
>>>>>> Some thoughts on my deployment
>>>>>>
>>>>>> *Logstash*
>>>>>> I think I’m going to need to re-introduce logstash just to leverage
>>>>>> the existing open source material of logstash filters and Kibana desktops.
>>>>>> VMware, ASA for example but wanted more real time data. I could
>>>>>> probably do the realtime tags with pattendb.
>>>>>>
>>>>>> *syslog-ng counters*
>>>>>> We use an IPAM API to create unique filters, log and destination conf
>>>>>> files. The goal was to get unique syslog counters for every VLAN realtime
>>>>>> directly from syslog-ng-ctl stats..
>>>>>>
>>>>>>
>>>>>> @include IPAM-filters
>>>>>> filter f_192_168_252_0 { netmask(192.168.252.0/24);};
>>>>>> filter f_192_168_253_0 { netmask(192.168.253.0/24);};
>>>>>> filter f_192_168_254_0 { netmask(192.168.254.0/30);};
>>>>>>
>>>>>>
>>>>>> @include IPAM-dest.conf
>>>>>> destination d_192_168_252_0 {
>>>>>> file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>>> destination d_192_168_253_0 {
>>>>>> file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>>> destination d_192_168_254_0 {
>>>>>> file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>>>
>>>>>> @include IPAM-log.conf
>>>>>> log { source(s_net); filter(f_192_168_252_0);
>>>>>> destination(d_192_168_252_0);};
>>>>>> log { source(s_net); filter(f_192_168_253_0);
>>>>>> destination(d_192_168_253_0);};
>>>>>> log { source(s_net); filter(f_192_168_254_0);
>>>>>> destination(d_192_168_254_0);};
>>>>>> log { source(s_net); filter(f_192_168_254_4);
>>>>>> destination(d_192_168_254_4);};
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn at gmail.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Does anyone have links or care to share notes on making a syslog-ng
>>>>>> -> ELK scale for enterprise ?
>>>>>>
>>>>>> I have some ideas and will gladly share my solution but also don’t
>>>>>> want to spend days figuring these things out that have already been built.
>>>>>> There are many ELK specific references but I also want to make sure
>>>>>> the model fits the syslog workload.
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ______________________________________________________________________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation:
>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> --
> Bazsi
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160421/7c6a3982/attachment.htm
More information about the syslog-ng
mailing list