[syslog-ng] ELK herd to scale
Czanik, Péter
peter.czanik at balabit.com
Thu Apr 21 15:06:02 CEST 2016
Hi,
ES2 support is only available in the not yet released syslog-ng OSE 3.8 (I
have some unofficial RPMs, but ES2 part is not yet tested, I plan to do it
next week). 3.7, which is the latest stable release, only supports ES1.
Yotu can name you index whatever you like. I named it "syslog-ng_..." as I
prefer to give unique names instead of the expected one. The more important
part is the ${YEAR}.${MONTH}.${DAY}" part, as Kibana expect the actual date
as well. So you can name it: "logstash-${YEAR}.${MONTH}.${DAY}" and
configure Kibana easier. It might also help you to use pre-built
dashboards, but I don't know, as I always built dashboards myself...
Bye,
Peter Czanik (CzP) <peter.czanik at balabit.com>
Balabit / syslog-ng upstream
http://czanik.blogs.balabit.com/
https://twitter.com/PCzanik
On Thu, Apr 21, 2016 at 1:51 PM, Scot Needy <scotrn at gmail.com> wrote:
> ES2 and Kibanna 4 but hold on a sec…
>
> I used your blog as a starting reference. Great job BTW, but I don’t see
> how the logstash index can exist using this directive in syslogng.conf
>
> option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}")
>
>
> On Apr 21, 2016, at 3:41 AM, Czanik, Péter <peter.czanik at balabit.com>
> wrote:
>
> Hi,
>
> To get started with syslog-ng + patterndb + ElasticSearch, you could use
> my blog at
> https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/
> It shows how to parse ssh log-in messages, and how to display them with
> Kibana.
>
> If you use "logstash-${YEAR}.${MONTH}.${DAY}" as index name, Kibana should
> find your logs without any extra setup.
>
> Which versions of Elasticsearch and Kibana do you use?
>
> Bye,
>
> Peter Czanik (CzP) <peter.czanik at balabit.com>
> Balabit / syslog-ng upstream
> http://czanik.blogs.balabit.com/
> https://twitter.com/PCzanik
>
> On Thu, Apr 21, 2016 at 9:13 AM, Balazs Scheidler <bazsi77 at gmail.com>
> wrote:
>
>> well, one thing that is probably a lot faster with syslog-ng is parsing,
>> so I guess it all boils down to performance needed at the reception side.
>>
>> receiving and writing out into a file, then polling that file for changes
>> is definitely slower if you have non-trival amount of messages. syslog-ng
>> receve & parse can happen at the 100k/second range and certainly ES is
>> usually slower than that, but that's the entire point in scaling, right? so
>> if I scale ES to the 100k/sec range, doing this with syslog-ng reduces the
>> load a lot at the data center collector a lot.
>>
>> On Thu, Apr 21, 2016 at 6:02 AM, Orangepeel Beef <
>> orangepeelbeef at gmail.com> wrote:
>>
>>> We used rsyslog to receive from all networking devices, and
>>> rsyslog<->rsyslog for systems. then I did some addtional processing of the
>>> logs for realtime alerting and shove them into ES with logstash. They all
>>> complement each other. Then you just need to stand up 1 log collection (in
>>> your case syslog-ng) server in each physical datacenter, and use logstash
>>> to parse the logs, and lumberjack to send them out to some central ES stack.
>>>
>>> On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn at gmail.com> wrote:
>>>
>>>>
>>>> That seems to be the way most people are doing it but I think in
>>>> absence of syslog-ng not in place of it.
>>>> But I don’t like the idea of waiting every XX minutes to see my ASA
>>>> logs.
>>>>
>>>> My understanding is the Logstash part of the ELK stack is not required
>>>> if you use the syslog-ng Elasticsearch plugin.
>>>> pro Realtime data
>>>> pro No additional hop for your data.
>>>>
>>>>
>>>> On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef at gmail.com>
>>>> wrote:
>>>>
>>>> They way I always liked doing it was to send all the logs via syslog
>>>> regularly to your central collection server and use logstash file input to
>>>> parse them in and shove into ES.
>>>>
>>>> On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <
>>>> orangepeelbeef at gmail.com> wrote:
>>>>
>>>>> logstash-* index is for logs that have been ingested via logstash of
>>>>> course :)
>>>>>
>>>>> every component of ELK scales horizontally extremely well.
>>>>>
>>>>>
>>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free.
>>>>> www.avast.com
>>>>> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
>>>>>
>>>>> On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn at gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template
>>>>>>
>>>>>> May have misspoke. Using ELK and patterndb.xml is new to me and I am
>>>>>> still trying to learn the mechanics.
>>>>>>
>>>>>>
>>>>>> I started by looking at Google for Kibana dashboard templates, one
>>>>>> of the better results here.
>>>>>> https://github.com/markwalkom/kibana-dashboards Most of the kibana
>>>>>> json templates I have seen on the net are setup for a logstash-* “index”
>>>>>> ?.
>>>>>>
>>>>>> I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So
>>>>>> time and ease of setup and support community size are big considerations. I
>>>>>> want to enable GeoIP for ASA data, NetFlow data and be able to leverage
>>>>>> existing templates logstash or patterndb for common applications. Apache,
>>>>>> Linux Syslog, Storage syslog, etc…
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <
>>>>>> balazs.scheidler at balabit.com> wrote:
>>>>>>
>>>>>> Can you pls point me to the direction of the logstash material you
>>>>>> mentioned? I would be interested in them whether it'd be possible to port
>>>>>> them over.
>>>>>> On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn at gmail.com> wrote:
>>>>>>
>>>>>>> Some thoughts on my deployment
>>>>>>>
>>>>>>> *Logstash*
>>>>>>> I think I’m going to need to re-introduce logstash just to leverage
>>>>>>> the existing open source material of logstash filters and Kibana desktops.
>>>>>>> VMware, ASA for example but wanted more real time data. I could
>>>>>>> probably do the realtime tags with pattendb.
>>>>>>>
>>>>>>> *syslog-ng counters*
>>>>>>> We use an IPAM API to create unique filters, log and destination
>>>>>>> conf files. The goal was to get unique syslog counters for every VLAN
>>>>>>> realtime directly from syslog-ng-ctl stats..
>>>>>>>
>>>>>>>
>>>>>>> @include IPAM-filters
>>>>>>> filter f_192_168_252_0 { netmask(192.168.252.0/24);};
>>>>>>> filter f_192_168_253_0 { netmask(192.168.253.0/24);};
>>>>>>> filter f_192_168_254_0 { netmask(192.168.254.0/30);};
>>>>>>>
>>>>>>>
>>>>>>> @include IPAM-dest.conf
>>>>>>> destination d_192_168_252_0 {
>>>>>>> file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>>>> destination d_192_168_253_0 {
>>>>>>> file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>>>> destination d_192_168_254_0 {
>>>>>>> file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
>>>>>>>
>>>>>>> @include IPAM-log.conf
>>>>>>> log { source(s_net); filter(f_192_168_252_0);
>>>>>>> destination(d_192_168_252_0);};
>>>>>>> log { source(s_net); filter(f_192_168_253_0);
>>>>>>> destination(d_192_168_253_0);};
>>>>>>> log { source(s_net); filter(f_192_168_254_0);
>>>>>>> destination(d_192_168_254_0);};
>>>>>>> log { source(s_net); filter(f_192_168_254_4);
>>>>>>> destination(d_192_168_254_4);};
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn at gmail.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Does anyone have links or care to share notes on making a syslog-ng
>>>>>>> -> ELK scale for enterprise ?
>>>>>>>
>>>>>>> I have some ideas and will gladly share my solution but also don’t
>>>>>>> want to spend days figuring these things out that have already been built.
>>>>>>> There are many ELK specific references but I also want to make sure
>>>>>>> the model fits the syslog workload.
>>>>>>>
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ______________________________________________________________________________
>>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>>> Documentation:
>>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ______________________________________________________________________________
>>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>> Documentation:
>>>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>>
>>>
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>>
>> --
>> Bazsi
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20160421/a5fbd323/attachment-0001.htm
More information about the syslog-ng
mailing list