Hi, Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ? I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload. Thanks
Some thoughts on my deployment Logstash I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb. syslog-ng counters We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats.. @include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24);}; filter f_192_168_253_0 { netmask(192.168.253.0/24);}; filter f_192_168_254_0 { netmask(192.168.254.0/30);}; @include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; @include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over. On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com> wrote:
Some thoughts on my deployment
*Logstash* I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
*syslog-ng counters* We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24);}; filter f_192_168_253_0 { netmask(192.168.253.0/24);}; filter f_192_168_254_0 { netmask(192.168.254.0/30);};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q... <https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q... dashboard template> May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics. I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards <https://github.com/markwalkom/kibana-dashboards> Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?. I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <balazs.scheidler@balabit.com> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over.
On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote: Some thoughts on my deployment
Logstash I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
syslog-ng counters We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24); <http://192.168.252.0/24);>}; filter f_192_168_253_0 { netmask(192.168.253.0/24); <http://192.168.253.0/24);>}; filter f_192_168_254_0 { netmask(192.168.254.0/30); <http://192.168.254.0/30);>};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
logstash-* index is for logs that have been ingested via logstash of course :) every component of ELK scales horizontally extremely well. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <#DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2> On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q...
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over. On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com> wrote:
Some thoughts on my deployment
*Logstash* I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
*syslog-ng counters* We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24);}; filter f_192_168_253_0 { netmask(192.168.253.0/24);}; filter f_192_168_254_0 { netmask(192.168.254.0/30);};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES. On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <#m_-2741954605978871063_DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q...
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over. On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com> wrote:
Some thoughts on my deployment
*Logstash* I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
*syslog-ng counters* We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24);}; filter f_192_168_253_0 { netmask(192.168.253.0/24);}; filter f_192_168_254_0 { netmask(192.168.254.0/30);};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs. My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. pro Realtime data pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote: logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <x-msg://2/#m_-2741954605978871063_DDB4FAA8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q... <https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template>
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards <https://github.com/markwalkom/kibana-dashboards> Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <balazs.scheidler@balabit.com <mailto:balazs.scheidler@balabit.com>> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over.
On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote: Some thoughts on my deployment
Logstash I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
syslog-ng counters We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24); <http://192.168.252.0/24);>}; filter f_192_168_253_0 { netmask(192.168.253.0/24); <http://192.168.253.0/24);>}; filter f_192_168_254_0 { netmask(192.168.254.0/30); <http://192.168.254.0/30);>};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
We used rsyslog to receive from all networking devices, and rsyslog<->rsyslog for systems. then I did some addtional processing of the logs for realtime alerting and shove them into ES with logstash. They all complement each other. Then you just need to stand up 1 log collection (in your case syslog-ng) server in each physical datacenter, and use logstash to parse the logs, and lumberjack to send them out to some central ES stack. On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn@gmail.com> wrote:
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs.
My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. pro Realtime data pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <orangepeelbeef@gmail.com
wrote:
logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q...
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over. On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com> wrote:
Some thoughts on my deployment
*Logstash* I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
*syslog-ng counters* We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24);}; filter f_192_168_253_0 { netmask(192.168.253.0/24);}; filter f_192_168_254_0 { netmask(192.168.254.0/30);};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Ingestion usually takes maybe a few seconds, so you're in near realtime in ELK. You can scale the log ingester by having multiple syslog-ng and logstash/lumberjack endpoints in each location once the processing starts to lag. Either using a LB or splitting your env. On Wed, Apr 20, 2016 at 9:02 PM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
We used rsyslog to receive from all networking devices, and rsyslog<->rsyslog for systems. then I did some addtional processing of the logs for realtime alerting and shove them into ES with logstash. They all complement each other. Then you just need to stand up 1 log collection (in your case syslog-ng) server in each physical datacenter, and use logstash to parse the logs, and lumberjack to send them out to some central ES stack.
On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn@gmail.com> wrote:
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs.
My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. pro Realtime data pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef < orangepeelbeef@gmail.com> wrote:
logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q...
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over. On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com> wrote:
Some thoughts on my deployment
*Logstash* I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
*syslog-ng counters* We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24);}; filter f_192_168_253_0 { netmask(192.168.253.0/24);}; filter f_192_168_254_0 { netmask(192.168.254.0/30);};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
well, one thing that is probably a lot faster with syslog-ng is parsing, so I guess it all boils down to performance needed at the reception side. receiving and writing out into a file, then polling that file for changes is definitely slower if you have non-trival amount of messages. syslog-ng receve & parse can happen at the 100k/second range and certainly ES is usually slower than that, but that's the entire point in scaling, right? so if I scale ES to the 100k/sec range, doing this with syslog-ng reduces the load a lot at the data center collector a lot. On Thu, Apr 21, 2016 at 6:02 AM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
We used rsyslog to receive from all networking devices, and rsyslog<->rsyslog for systems. then I did some addtional processing of the logs for realtime alerting and shove them into ES with logstash. They all complement each other. Then you just need to stand up 1 log collection (in your case syslog-ng) server in each physical datacenter, and use logstash to parse the logs, and lumberjack to send them out to some central ES stack.
On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn@gmail.com> wrote:
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs.
My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. pro Realtime data pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef < orangepeelbeef@gmail.com> wrote:
logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q...
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over. On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com> wrote:
Some thoughts on my deployment
*Logstash* I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
*syslog-ng counters* We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24);}; filter f_192_168_253_0 { netmask(192.168.253.0/24);}; filter f_192_168_254_0 { netmask(192.168.254.0/30);};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
Hi, To get started with syslog-ng + patterndb + ElasticSearch, you could use my blog at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-st... It shows how to parse ssh log-in messages, and how to display them with Kibana. If you use "logstash-${YEAR}.${MONTH}.${DAY}" as index name, Kibana should find your logs without any extra setup. Which versions of Elasticsearch and Kibana do you use? Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Thu, Apr 21, 2016 at 9:13 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
well, one thing that is probably a lot faster with syslog-ng is parsing, so I guess it all boils down to performance needed at the reception side.
receiving and writing out into a file, then polling that file for changes is definitely slower if you have non-trival amount of messages. syslog-ng receve & parse can happen at the 100k/second range and certainly ES is usually slower than that, but that's the entire point in scaling, right? so if I scale ES to the 100k/sec range, doing this with syslog-ng reduces the load a lot at the data center collector a lot.
On Thu, Apr 21, 2016 at 6:02 AM, Orangepeel Beef <orangepeelbeef@gmail.com
wrote:
We used rsyslog to receive from all networking devices, and rsyslog<->rsyslog for systems. then I did some addtional processing of the logs for realtime alerting and shove them into ES with logstash. They all complement each other. Then you just need to stand up 1 log collection (in your case syslog-ng) server in each physical datacenter, and use logstash to parse the logs, and lumberjack to send them out to some central ES stack.
On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn@gmail.com> wrote:
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs.
My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. pro Realtime data pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef < orangepeelbeef@gmail.com> wrote:
logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q...
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over. On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com> wrote:
Some thoughts on my deployment
*Logstash* I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
*syslog-ng counters* We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24);}; filter f_192_168_253_0 { netmask(192.168.253.0/24);}; filter f_192_168_254_0 { netmask(192.168.254.0/30);};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
ES2 and Kibanna 4 but hold on a sec… I used your blog as a starting reference. Great job BTW, but I don’t see how the logstash index can exist using this directive in syslogng.conf option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}")
On Apr 21, 2016, at 3:41 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
To get started with syslog-ng + patterndb + ElasticSearch, you could use my blog at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-st... <https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/> It shows how to parse ssh log-in messages, and how to display them with Kibana.
If you use "logstash-${YEAR}.${MONTH}.${DAY}" as index name, Kibana should find your logs without any extra setup.
Which versions of Elasticsearch and Kibana do you use?
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ <http://czanik.blogs.balabit.com/> https://twitter.com/PCzanik <https://twitter.com/PCzanik> On Thu, Apr 21, 2016 at 9:13 AM, Balazs Scheidler <bazsi77@gmail.com <mailto:bazsi77@gmail.com>> wrote: well, one thing that is probably a lot faster with syslog-ng is parsing, so I guess it all boils down to performance needed at the reception side.
receiving and writing out into a file, then polling that file for changes is definitely slower if you have non-trival amount of messages. syslog-ng receve & parse can happen at the 100k/second range and certainly ES is usually slower than that, but that's the entire point in scaling, right? so if I scale ES to the 100k/sec range, doing this with syslog-ng reduces the load a lot at the data center collector a lot.
On Thu, Apr 21, 2016 at 6:02 AM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote: We used rsyslog to receive from all networking devices, and rsyslog<->rsyslog for systems. then I did some addtional processing of the logs for realtime alerting and shove them into ES with logstash. They all complement each other. Then you just need to stand up 1 log collection (in your case syslog-ng) server in each physical datacenter, and use logstash to parse the logs, and lumberjack to send them out to some central ES stack.
On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs.
My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. pro Realtime data pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote: logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q... <https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template>
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards <https://github.com/markwalkom/kibana-dashboards> Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <balazs.scheidler@balabit.com <mailto:balazs.scheidler@balabit.com>> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over.
On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote: Some thoughts on my deployment
Logstash I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
syslog-ng counters We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24); <http://192.168.252.0/24);>}; filter f_192_168_253_0 { netmask(192.168.253.0/24); <http://192.168.253.0/24);>}; filter f_192_168_254_0 { netmask(192.168.254.0/30); <http://192.168.254.0/30);>};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, ES2 support is only available in the not yet released syslog-ng OSE 3.8 (I have some unofficial RPMs, but ES2 part is not yet tested, I plan to do it next week). 3.7, which is the latest stable release, only supports ES1. Yotu can name you index whatever you like. I named it "syslog-ng_..." as I prefer to give unique names instead of the expected one. The more important part is the ${YEAR}.${MONTH}.${DAY}" part, as Kibana expect the actual date as well. So you can name it: "logstash-${YEAR}.${MONTH}.${DAY}" and configure Kibana easier. It might also help you to use pre-built dashboards, but I don't know, as I always built dashboards myself... Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik On Thu, Apr 21, 2016 at 1:51 PM, Scot Needy <scotrn@gmail.com> wrote:
ES2 and Kibanna 4 but hold on a sec…
I used your blog as a starting reference. Great job BTW, but I don’t see how the logstash index can exist using this directive in syslogng.conf
option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}")
On Apr 21, 2016, at 3:41 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
To get started with syslog-ng + patterndb + ElasticSearch, you could use my blog at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-st... It shows how to parse ssh log-in messages, and how to display them with Kibana.
If you use "logstash-${YEAR}.${MONTH}.${DAY}" as index name, Kibana should find your logs without any extra setup.
Which versions of Elasticsearch and Kibana do you use?
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ https://twitter.com/PCzanik
On Thu, Apr 21, 2016 at 9:13 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
well, one thing that is probably a lot faster with syslog-ng is parsing, so I guess it all boils down to performance needed at the reception side.
receiving and writing out into a file, then polling that file for changes is definitely slower if you have non-trival amount of messages. syslog-ng receve & parse can happen at the 100k/second range and certainly ES is usually slower than that, but that's the entire point in scaling, right? so if I scale ES to the 100k/sec range, doing this with syslog-ng reduces the load a lot at the data center collector a lot.
On Thu, Apr 21, 2016 at 6:02 AM, Orangepeel Beef < orangepeelbeef@gmail.com> wrote:
We used rsyslog to receive from all networking devices, and rsyslog<->rsyslog for systems. then I did some addtional processing of the logs for realtime alerting and shove them into ES with logstash. They all complement each other. Then you just need to stand up 1 log collection (in your case syslog-ng) server in each physical datacenter, and use logstash to parse the logs, and lumberjack to send them out to some central ES stack.
On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn@gmail.com> wrote:
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs.
My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. pro Realtime data pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef < orangepeelbeef@gmail.com> wrote:
logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q...
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over. On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com> wrote:
> Some thoughts on my deployment > > *Logstash* > I think I’m going to need to re-introduce logstash just to leverage > the existing open source material of logstash filters and Kibana desktops. > VMware, ASA for example but wanted more real time data. I could > probably do the realtime tags with pattendb. > > *syslog-ng counters* > We use an IPAM API to create unique filters, log and destination > conf files. The goal was to get unique syslog counters for every VLAN > realtime directly from syslog-ng-ctl stats.. > > > @include IPAM-filters > filter f_192_168_252_0 { netmask(192.168.252.0/24);}; > filter f_192_168_253_0 { netmask(192.168.253.0/24);}; > filter f_192_168_254_0 { netmask(192.168.254.0/30);}; > > > @include IPAM-dest.conf > destination d_192_168_252_0 { > file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; > destination d_192_168_253_0 { > file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; > destination d_192_168_254_0 { > file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; > > @include IPAM-log.conf > log { source(s_net); filter(f_192_168_252_0); > destination(d_192_168_252_0);}; > log { source(s_net); filter(f_192_168_253_0); > destination(d_192_168_253_0);}; > log { source(s_net); filter(f_192_168_254_0); > destination(d_192_168_254_0);}; > log { source(s_net); filter(f_192_168_254_4); > destination(d_192_168_254_4);}; > > > > On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote: > > > > Hi, > > Does anyone have links or care to share notes on making a syslog-ng > -> ELK scale for enterprise ? > > I have some ideas and will gladly share my solution but also don’t > want to spend days figuring these things out that have already been built. > There are many ELK specific references but I also want to make sure > the model fits the syslog workload. > > > Thanks > > > > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: > http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Yes, It is “unsupported” but I did not want to start out completing the deployment behind a release. I think its getting a bit off topic. The decision I am trying to make is on logstash. Without it I seem to loose the ability to use a majority of the public templates out there for Kibanna dashboards and log filter templates from the logstash community. With logstash I need to daisy chain this log management process to import data and my ASA logs are always behind. I still have a couple of weeks of testing and tuning and figuring out stuff before I start my chef cookbooks. CentOS Linux release 7.2.1511 ISO elasticsearch-2.3.1 from RPM kibana-4.5.0-linux-x64 from DOWNLOAD syslog-ng 3.8.0alpha0 from GIT Installer-Version: 3.8.0alpha0 Revision: Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: syslogformat,afsocket,affile,afprog,afuser,afamqp,afmongodb,afsmtp,csvparser,confgen,system-source,linux-kmsg-format,basicfuncs,cryptofuncs,dbparser,json-plugin,geoip-plugin,afstomp,pseudofile,graphite,sdjournal,mod-java,kvformat,date,cef,disk-buffer Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: off
On Apr 21, 2016, at 9:06 AM, Czanik, Péter <peter.czanik@balabit.com> wrote:
Hi,
ES2 support is only available in the not yet released syslog-ng OSE 3.8 (I have some unofficial RPMs, but ES2 part is not yet tested, I plan to do it next week). 3.7, which is the latest stable release, only supports ES1.
Yotu can name you index whatever you like. I named it "syslog-ng_..." as I prefer to give unique names instead of the expected one. The more important part is the ${YEAR}.${MONTH}.${DAY}" part, as Kibana expect the actual date as well. So you can name it: "logstash-${YEAR}.${MONTH}.${DAY}" and configure Kibana easier. It might also help you to use pre-built dashboards, but I don't know, as I always built dashboards myself...
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ <http://czanik.blogs.balabit.com/> https://twitter.com/PCzanik <https://twitter.com/PCzanik> On Thu, Apr 21, 2016 at 1:51 PM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote: ES2 and Kibanna 4 but hold on a sec…
I used your blog as a starting reference. Great job BTW, but I don’t see how the logstash index can exist using this directive in syslogng.conf
option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}")
On Apr 21, 2016, at 3:41 AM, Czanik, Péter <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
Hi,
To get started with syslog-ng + patterndb + ElasticSearch, you could use my blog at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-st... <https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/> It shows how to parse ssh log-in messages, and how to display them with Kibana.
If you use "logstash-${YEAR}.${MONTH}.${DAY}" as index name, Kibana should find your logs without any extra setup.
Which versions of Elasticsearch and Kibana do you use?
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ <http://czanik.blogs.balabit.com/> https://twitter.com/PCzanik <https://twitter.com/PCzanik> On Thu, Apr 21, 2016 at 9:13 AM, Balazs Scheidler <bazsi77@gmail.com <mailto:bazsi77@gmail.com>> wrote: well, one thing that is probably a lot faster with syslog-ng is parsing, so I guess it all boils down to performance needed at the reception side.
receiving and writing out into a file, then polling that file for changes is definitely slower if you have non-trival amount of messages. syslog-ng receve & parse can happen at the 100k/second range and certainly ES is usually slower than that, but that's the entire point in scaling, right? so if I scale ES to the 100k/sec range, doing this with syslog-ng reduces the load a lot at the data center collector a lot.
On Thu, Apr 21, 2016 at 6:02 AM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote: We used rsyslog to receive from all networking devices, and rsyslog<->rsyslog for systems. then I did some addtional processing of the logs for realtime alerting and shove them into ES with logstash. They all complement each other. Then you just need to stand up 1 log collection (in your case syslog-ng) server in each physical datacenter, and use logstash to parse the logs, and lumberjack to send them out to some central ES stack.
On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs.
My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. pro Realtime data pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote: logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q... <https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template>
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards <https://github.com/markwalkom/kibana-dashboards> Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <balazs.scheidler@balabit.com <mailto:balazs.scheidler@balabit.com>> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over.
On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote: Some thoughts on my deployment
Logstash I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
syslog-ng counters We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24); <http://192.168.252.0/24);>}; filter f_192_168_253_0 { netmask(192.168.253.0/24); <http://192.168.253.0/24);>}; filter f_192_168_254_0 { netmask(192.168.254.0/30); <http://192.168.254.0/30);>};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I was mistaken on logstash. It appears that it has a TCP/UDP input. Has anyone done this with success ? syslog-ng -> TCP -> logstashInput -> logstashFormat -> logstashOutput -> es2.
On Apr 21, 2016, at 10:33 AM, Scot Needy <scotrn@gmail.com> wrote:
Yes, It is “unsupported” but I did not want to start out completing the deployment behind a release.
I think its getting a bit off topic. The decision I am trying to make is on logstash.
Without it I seem to loose the ability to use a majority of the public templates out there for Kibanna dashboards and log filter templates from the logstash community. With logstash I need to daisy chain this log management process to import data and my ASA logs are always behind.
I still have a couple of weeks of testing and tuning and figuring out stuff before I start my chef cookbooks.
CentOS Linux release 7.2.1511 ISO elasticsearch-2.3.1 from RPM kibana-4.5.0-linux-x64 from DOWNLOAD syslog-ng 3.8.0alpha0 from GIT Installer-Version: 3.8.0alpha0 Revision: Module-Directory: /opt/syslog-ng/lib/syslog-ng Module-Path: /opt/syslog-ng/lib/syslog-ng Available-Modules: syslogformat,afsocket,affile,afprog,afuser,afamqp,afmongodb,afsmtp,csvparser,confgen,system-source,linux-kmsg-format,basicfuncs,cryptofuncs,dbparser,json-plugin,geoip-plugin,afstomp,pseudofile,graphite,sdjournal,mod-java,kvformat,date,cef,disk-buffer Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: off
On Apr 21, 2016, at 9:06 AM, Czanik, Péter <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
Hi,
ES2 support is only available in the not yet released syslog-ng OSE 3.8 (I have some unofficial RPMs, but ES2 part is not yet tested, I plan to do it next week). 3.7, which is the latest stable release, only supports ES1.
Yotu can name you index whatever you like. I named it "syslog-ng_..." as I prefer to give unique names instead of the expected one. The more important part is the ${YEAR}.${MONTH}.${DAY}" part, as Kibana expect the actual date as well. So you can name it: "logstash-${YEAR}.${MONTH}.${DAY}" and configure Kibana easier. It might also help you to use pre-built dashboards, but I don't know, as I always built dashboards myself...
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ <http://czanik.blogs.balabit.com/> https://twitter.com/PCzanik <https://twitter.com/PCzanik> On Thu, Apr 21, 2016 at 1:51 PM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote: ES2 and Kibanna 4 but hold on a sec…
I used your blog as a starting reference. Great job BTW, but I don’t see how the logstash index can exist using this directive in syslogng.conf
option("index", "syslog-ng_${YEAR}.${MONTH}.${DAY}")
On Apr 21, 2016, at 3:41 AM, Czanik, Péter <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> wrote:
Hi,
To get started with syslog-ng + patterndb + ElasticSearch, you could use my blog at https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-st... <https://czanik.blogs.balabit.com/2015/10/how-to-parse-data-with-syslog-ng-store-in-elasticsearch-and-analyze-with-kibana/> It shows how to parse ssh log-in messages, and how to display them with Kibana.
If you use "logstash-${YEAR}.${MONTH}.${DAY}" as index name, Kibana should find your logs without any extra setup.
Which versions of Elasticsearch and Kibana do you use?
Bye,
Peter Czanik (CzP) <peter.czanik@balabit.com <mailto:peter.czanik@balabit.com>> Balabit / syslog-ng upstream http://czanik.blogs.balabit.com/ <http://czanik.blogs.balabit.com/> https://twitter.com/PCzanik <https://twitter.com/PCzanik> On Thu, Apr 21, 2016 at 9:13 AM, Balazs Scheidler <bazsi77@gmail.com <mailto:bazsi77@gmail.com>> wrote: well, one thing that is probably a lot faster with syslog-ng is parsing, so I guess it all boils down to performance needed at the reception side.
receiving and writing out into a file, then polling that file for changes is definitely slower if you have non-trival amount of messages. syslog-ng receve & parse can happen at the 100k/second range and certainly ES is usually slower than that, but that's the entire point in scaling, right? so if I scale ES to the 100k/sec range, doing this with syslog-ng reduces the load a lot at the data center collector a lot.
On Thu, Apr 21, 2016 at 6:02 AM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote: We used rsyslog to receive from all networking devices, and rsyslog<->rsyslog for systems. then I did some addtional processing of the logs for realtime alerting and shove them into ES with logstash. They all complement each other. Then you just need to stand up 1 log collection (in your case syslog-ng) server in each physical datacenter, and use logstash to parse the logs, and lumberjack to send them out to some central ES stack.
On Wed, Apr 20, 2016 at 8:26 PM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs.
My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin. pro Realtime data pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <orangepeelbeef@gmail.com <mailto:orangepeelbeef@gmail.com>> wrote: logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link> <>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q... <https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q=kibana%20dashboard%20template>
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards <https://github.com/markwalkom/kibana-dashboards> Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs <balazs.scheidler@balabit.com <mailto:balazs.scheidler@balabit.com>> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over.
On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote: Some thoughts on my deployment
Logstash I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
syslog-ng counters We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24); <http://192.168.252.0/24);>}; filter f_192_168_253_0 { netmask(192.168.253.0/24); <http://192.168.253.0/24);>}; filter f_192_168_254_0 { netmask(192.168.254.0/30); <http://192.168.254.0/30);>};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com <mailto:scotrn@gmail.com>> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
-- Bazsi
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng <https://lists.balabit.hu/mailman/listinfo/syslog-ng> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng <http://www.balabit.com/support/documentation/?product=syslog-ng> FAQ: http://www.balabit.com/wiki/syslog-ng-faq <http://www.balabit.com/wiki/syslog-ng-faq>
I do: syslog-ng > file > logstash-file-input > logstash-massaging > ES From: <syslog-ng-bounces@lists.balabit.hu> on behalf of Scot Needy <scotrn@gmail.com> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Date: Friday, April 22, 2016 at 09:35 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] ELK herd to scale I was mistaken on logstash. It appears that it has a TCP/UDP input. Has anyone done this with success ? syslog-ng -> TCP -> logstashInput -> logstashFormat -> logstashOutput -> es2. -------------------------------------------------------------------------------------------------------------------------- This email has been sent to you on behalf of Nephila Advisors LLC (“Advisors”). Advisors provides consultancy services to Nephila Capital Ltd. (“Capital”), an investment advisor managed and carrying on business in Bermuda. Advisors and its employees do not act as agents for Capital or the funds it advises and do not have the authority to bind Capital or such funds to any transaction or agreement. The information in this e-mail, and any attachment therein, is confidential and for use by the addressee only. Any use, disclosure, reproduction, modification or distribution of the contents of this e-mail, or any part thereof, other than by the intended recipient, is strictly prohibited. If you are not the intended recipient, please return the e-mail to the sender and delete it from your computer. This email is for information purposes only, nothing contained herein constitutes an offer to sell or buy securities, as such an offer may only be made from a properly authorized offering document. Although Nephila attempts to sweep e-mail and attachments for viruses, it does not guarantee that either are virus-free and accepts no liability for any damage sustained as a result of viruses. --------------------------------------------------------------------------------------------------------------------------
Hi, On Thu, Apr 21, 2016 at 5:26 AM, Scot Needy <scotrn@gmail.com> wrote:
That seems to be the way most people are doing it but I think in absence of syslog-ng not in place of it. But I don’t like the idea of waiting every XX minutes to see my ASA logs.
My understanding is the Logstash part of the ELK stack is not required if you use the syslog-ng Elasticsearch plugin.
yes, that's the point of the syslog-ng Elasticseach destination: you can build an ESK stack (Elasticsearch/syslog-ng/Kibana). pro Realtime data
pro No additional hop for your data.
On Apr 20, 2016, at 10:46 PM, Orangepeel Beef <orangepeelbeef@gmail.com> wrote:
They way I always liked doing it was to send all the logs via syslog regularly to your central collection server and use logstash file input to parse them in and shove into ES.
On Wed, Apr 20, 2016 at 7:43 PM, Orangepeel Beef <orangepeelbeef@gmail.com
wrote:
logstash-* index is for logs that have been ingested via logstash of course :)
every component of ELK scales horizontally extremely well.
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=icon> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail&utm_term=link>
On Wed, Apr 20, 2016 at 12:41 PM, Scot Needy <scotrn@gmail.com> wrote:
https://www.google.com/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=UTF-8#q...
May have misspoke. Using ELK and patterndb.xml is new to me and I am still trying to learn the mechanics.
I started by looking at Google for Kibana dashboard templates, one of the better results here. https://github.com/markwalkom/kibana-dashboards Most of the kibana json templates I have seen on the net are setup for a logstash-* “index” ?.
I’m trying to set Syslog-ng-> ELK up in my “spare time” at work. So time and ease of setup and support community size are big considerations. I want to enable GeoIP for ASA data, NetFlow data and be able to leverage existing templates logstash or patterndb for common applications. Apache, Linux Syslog, Storage syslog, etc…
On Apr 20, 2016, at 2:13 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
Can you pls point me to the direction of the logstash material you mentioned? I would be interested in them whether it'd be possible to port them over. On Apr 20, 2016 7:00 PM, "Scot Needy" <scotrn@gmail.com> wrote:
Some thoughts on my deployment
*Logstash* I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
*syslog-ng counters* We use an IPAM API to create unique filters, log and destination conf files. The goal was to get unique syslog counters for every VLAN realtime directly from syslog-ng-ctl stats..
@include IPAM-filters filter f_192_168_252_0 { netmask(192.168.252.0/24);}; filter f_192_168_253_0 { netmask(192.168.253.0/24);}; filter f_192_168_254_0 { netmask(192.168.254.0/30);};
@include IPAM-dest.conf destination d_192_168_252_0 { file(/opt/syslog-ng/logs/192_168_252_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_253_0 { file(/opt/syslog-ng/logs/192_168_253_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);}; destination d_192_168_254_0 { file(/opt/syslog-ng/logs/192_168_254_0/$YEAR$MONTH$DAY-$HOUR-$HOST.log);};
@include IPAM-log.conf log { source(s_net); filter(f_192_168_252_0); destination(d_192_168_252_0);}; log { source(s_net); filter(f_192_168_253_0); destination(d_192_168_253_0);}; log { source(s_net); filter(f_192_168_254_0); destination(d_192_168_254_0);}; log { source(s_net); filter(f_192_168_254_4); destination(d_192_168_254_4);};
On Apr 20, 2016, at 11:18 AM, Scot Needy <scotrn@gmail.com> wrote:
Hi,
Does anyone have links or care to share notes on making a syslog-ng -> ELK scale for enterprise ?
I have some ideas and will gladly share my solution but also don’t want to spend days figuring these things out that have already been built. There are many ELK specific references but I also want to make sure the model fits the syslog workload.
Thanks
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi Scot, On Wed, Apr 20, 2016 at 01:00:26PM -0400, Scot Needy wrote:
Logstash I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
Just so you know, there actually is a grok parser in the incubator so this could help you migrate to syslog-ng.
Not sure if this thread is going the right way as there are several ways to accomplish the end goal of ES2 indexes. I am already using syslog-ng the question is on filtering and tagging platforms. syslog-ng Can parse streams and accomplish this in read time Can input directly to es2 eliminating a 3rd stage of processing Logstash Broader range of user contributed filters Can also work on a stream but not a replacement functionality for syslog. Standardized filter, tagging platform. If you are going to use logstash for other solutions then doesn’t it make sense to use it for all.
On Apr 25, 2016, at 5:16 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
Hi Scot,
On Wed, Apr 20, 2016 at 01:00:26PM -0400, Scot Needy wrote:
Logstash I think I’m going to need to re-introduce logstash just to leverage the existing open source material of logstash filters and Kibana desktops. VMware, ASA for example but wanted more real time data. I could probably do the realtime tags with pattendb.
Just so you know, there actually is a grok parser in the incubator so this could help you migrate to syslog-ng.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (8)
-
Balazs Scheidler
-
Budai, László
-
Czanik, Péter
-
Fabien Wernli
-
Orangepeel Beef
-
Robin Blanchard
-
Scheidler, Balázs
-
Scot Needy