Convert logstash filters to syslog-ng
Hi all, I would like to drop Logstash collector from our ELK infrastructure and use syslog-ng instead. This ELK infrastructure collects, report and show dashboards about security devices: firewalls, anti-spam devices, etc. Most of these logs arrives from rsyslog collectors (deployed in several linux and BSD machines). I have seen in Balabit's blog page how this could be done: https://www.balabit.com/blog/how-to-parse-data-with-syslog-ng-store-in-elast... and https://www.balabit.com/blog/collecting-and-parsing-suricata-logs-using-sysl.... The most important point here is to test all configured logstash filters inside syslog-ng: GeoIP patterns, some substitution params, etc. Any tips or tricks to accomplish this type of change? Many thanks. -- Greetings, C. L. Martinez
Hi Martinez You need to install syslog-ng first. Then create configuration file for your linux and BSD machines inside /etc/syslog-ng/conf.d Say you have configured one linux machine that forwards logs to this centralized syslog-ng server then you can use following configuration so that logs gets transferred to kibana. Forwarder (Linux machine) -> Syslog-ng ( Centralize log collector) -> Elastic search -> Kibana One such example can be like. # vi /etc/syslog-ng/conf.d/remote-linux-1.conf source s_2514 { tcp(port(2514)); }; # Assuming remote linux machine forwarding logs to syslog-ng server's tcp port 2514. destination d_remote_linux1 { tcp("127.0.0.1" port(9200) template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; # Change the above IP -> 127.0.0.1 to your elastic server's IP filter f_remote_linux1 { '' your filter " ; }; # Apply whatever filter you want, you can use multiple filters as well. log { source(s_2514); filter(f_remote_linux1); destination(d_remote_linux1); }; To use GeoIP in the above configuration, check this https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-o... I am not expert on ELK, so from elasticsearch to kibana, you need to find out - how it should be. Regards Dwijadas Dey On Mon, Apr 24, 2017 at 7:12 PM, C. L. Martinez <carlopmart@gmail.com> wrote:
Hi all,
I would like to drop Logstash collector from our ELK infrastructure and use syslog-ng instead. This ELK infrastructure collects, report and show dashboards about security devices: firewalls, anti-spam devices, etc.
Most of these logs arrives from rsyslog collectors (deployed in several linux and BSD machines). I have seen in Balabit's blog page how this could be done: https://www.balabit.com/blog/how-to-parse-data-with-syslog- ng-store-in-elasticsearch-and-analyze-with-kibana/ and https://www.balabit.com/blog/collecting-and-parsing- suricata-logs-using-syslog-ng/.
The most important point here is to test all configured logstash filters inside syslog-ng: GeoIP patterns, some substitution params, etc. Any tips or tricks to accomplish this type of change?
Many thanks.
-- Greetings, C. L. Martinez ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Hi, On Mon, Apr 24, 2017 at 01:42:43PM +0000, C. L. Martinez wrote:
The most important point here is to test all configured logstash filters inside syslog-ng: GeoIP patterns, some substitution params, etc. Any tips or tricks to accomplish this type of change?
If you have a lot of grok patterns, you might want to look at the grok parser in syslog-ng-incubator, which will let you use your existing rules out of the box. In the long term you will probably want to convert them to patterndb rules, the documentation of which is very complete. Cheers
Hi, Could you share your Logstash configuration with us? (At least the part which can be anonimized) I don't have much Logstash experience, but can help you figuring out which are the corresponding syslog-ng features. In the end it could be used as a Logstash to syslog-ng guide. Bye, Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://www.balabit.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Mon, Apr 24, 2017 at 3:42 PM, C. L. Martinez <carlopmart@gmail.com> wrote:
Hi all,
I would like to drop Logstash collector from our ELK infrastructure and use syslog-ng instead. This ELK infrastructure collects, report and show dashboards about security devices: firewalls, anti-spam devices, etc.
Most of these logs arrives from rsyslog collectors (deployed in several linux and BSD machines). I have seen in Balabit's blog page how this could be done: https://www.balabit.com/blog/how-to-parse-data-with-syslog- ng-store-in-elasticsearch-and-analyze-with-kibana/ and https://www.balabit.com/blog/collecting-and-parsing- suricata-logs-using-syslog-ng/.
The most important point here is to test all configured logstash filters inside syslog-ng: GeoIP patterns, some substitution params, etc. Any tips or tricks to accomplish this type of change?
Many thanks.
-- Greetings, C. L. Martinez ____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
C. L. Martinez
-
Czanik, Péter
-
Dwijadas Dey
-
Fabien Wernli