Sorry if I'm missing this but whats the best way to implement a json filter like these in syslog-ng, Patterndb? This is my logstash filter that sends data to a specific syslog-ng PORT for each condition which may grow and become a management headache when there are 6 logstash hosts that will need to support a filter for each breakout needed. I'd like to just send everything to a single syslog-ng port and have syslog-ng do the logic. which would then become. input { beats { port => 5044 } } output{ tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } *Logstash bloated output filters. * output{ if [type]=="wineventlog" and "DC" in [tags] { tcp { host => "loghost" port => "5142" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" and "PCI" in [tags] { tcp { host => "loghost" port => "5141" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" { tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "apache" in [tags] { tcp { host => "loghost" port => "5145" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "PCI" in [tags] { tcp { host => "loghost" port => "5144" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" { tcp { host => "loghost" port => "5143" mode => "client" codec => "json_lines" } } else { file { path => "/opt/syslog-ng/logs/logstash/%{host}-%{+YYYY-MM-dd}.json" codec => "json_lines" } } }
You can parse json using the json-parser() and filter any json name value pair simply with a filter expression. On Apr 3, 2018 17:39, "Scot" <scotrn@gmail.com> wrote:
Sorry if I'm missing this but whats the best way to implement a json filter like these in syslog-ng, Patterndb?
This is my logstash filter that sends data to a specific syslog-ng PORT for each condition which may grow and become a management headache when there are 6 logstash hosts that will need to support a filter for each breakout needed.
I'd like to just send everything to a single syslog-ng port and have syslog-ng do the logic. which would then become.
input { beats { port => 5044 } }
output{ tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } }
*Logstash bloated output filters. *
output{ if [type]=="wineventlog" and "DC" in [tags] { tcp { host => "loghost" port => "5142" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" and "PCI" in [tags] { tcp { host => "loghost" port => "5141" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" { tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "apache" in [tags] { tcp { host => "loghost" port => "5145" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "PCI" in [tags] { tcp { host => "loghost" port => "5144" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" { tcp { host => "loghost" port => "5143" mode => "client" codec => "json_lines" } } else { file { path => "/opt/syslog-ng/logs/logstash/%{host}-%{+YYYY-MM-dd}.json" codec => "json_lines" } } }
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
So then my log statement where I DON'T want duplicate copies would look something like. filter f_wineventlog_DC { "${type} eq "wineventlog" and "${tag1} eq "DC" }; filter f_wineventlog_PCI { "${type} eq "wineventlog" and "${tag1} eq "PCI" }; log { source(s_logstash); parser {json-parser();}; filter { f_wineventlog_DC(); }; destination(d_wineventlog_DC); log { filter("example"); destination(d_file2); }; };
output{
if [type]=="wineventlog" and "DC" in [tags] { tcp { host => "loghost" port => "5142" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" and "PCI" in [tags] { tcp { host => "loghost" port => "5141" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" { tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "apache" in [tags] { tcp { host => "loghost" port => "5145" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "PCI" in [tags] { tcp { host => "loghost" port => "5144" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" { tcp { host => "loghost" port => "5143" mode => "client" codec => "json_lines" } } else { file { path => "/opt/syslog-ng/logs/logstash/%{host}-%{+YYYY-MM-dd}.json" codec => "json_lines" } } }
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Sent before it's time. So then my log statement where I DON'T want duplicate copies would look something like this ? filter f_wineventlog_DC { "${type} eq "wineventlog" and "${tag1} eq "DC" }; filter f_wineventlog_PCI { "${type} eq "wineventlog" and "${tag1} eq "PCI" }; filter f_wineventlog { "${type} eq "wineventlog" }; log { source(s_logstash); parser {json-parser();}; filter(f_wineventlog_DC); destination(d_file1); log { filter(f_wineventlog_PCI); destination(d_file2); }; log { filter(f_wineventlog); destination(d_file3); }; }; On Tue, Apr 3, 2018 at 3:25 PM, Scot <scotrn@gmail.com> wrote:
So then my log statement where I DON'T want duplicate copies would look something like.
filter f_wineventlog_DC { "${type} eq "wineventlog" and "${tag1} eq "DC" }; filter f_wineventlog_PCI { "${type} eq "wineventlog" and "${tag1} eq "PCI" };
log { source(s_logstash);
parser {json-parser();};
filter { f_wineventlog_DC(); };
destination(d_wineventlog_DC);
log { filter("example"); destination(d_file2); };
};
output{
if [type]=="wineventlog" and "DC" in [tags] { tcp { host => "loghost" port => "5142" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" and "PCI" in [tags] { tcp { host => "loghost" port => "5141" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" { tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "apache" in [tags] { tcp { host => "loghost" port => "5145" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "PCI" in [tags] { tcp { host => "loghost" port => "5144" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" { tcp { host => "loghost" port => "5143" mode => "client" codec => "json_lines" } } else { file { path => "/opt/syslog-ng/logs/logstash/%{host}-%{+YYYY-MM-dd}.json" codec => "json_lines" } } }
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
If you add flags(final) to each of the log statements you wont get duplication. Also, master already has support for the if statement, that should be released in 3.15. if ("$type" = "DC") { destination { file(...); }; } elif (...) { ... } else { }; Thats more readable and achieves roughly the same. On Apr 3, 2018 21:25, "Scot" <scotrn@gmail.com> wrote:
So then my log statement where I DON'T want duplicate copies would look something like.
filter f_wineventlog_DC { "${type} eq "wineventlog" and "${tag1} eq "DC" }; filter f_wineventlog_PCI { "${type} eq "wineventlog" and "${tag1} eq "PCI" };
log { source(s_logstash);
parser {json-parser();};
filter { f_wineventlog_DC(); };
destination(d_wineventlog_DC);
log { filter("example"); destination(d_file2); };
};
output{
if [type]=="wineventlog" and "DC" in [tags] { tcp { host => "loghost" port => "5142" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" and "PCI" in [tags] { tcp { host => "loghost" port => "5141" mode => "client" codec => "json_lines" } } else if [type]=="wineventlog" { tcp { host => "loghost" port => "5140" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "apache" in [tags] { tcp { host => "loghost" port => "5145" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" and "PCI" in [tags] { tcp { host => "loghost" port => "5144" mode => "client" codec => "json_lines" } } else if [type]=="filebeat" { tcp { host => "loghost" port => "5143" mode => "client" codec => "json_lines" } } else { file { path => "/opt/syslog-ng/logs/logstash/%{host}-%{+YYYY-MM-dd}.json" codec => "json_lines" } } }
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (3)
-
Balazs Scheidler
-
Scheidler, Balázs
-
Scot