3.11 Multiple elastic destinations.
Has anyone had success/failure using multiple ES destinations in syslog-ng. I am want to direct traffic to different indexes based on syslog-ng filters destination d_es { elasticsearch2( client-mode("http") index("syslog-ng_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") cluster-url("http://elasticsearch:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("clustername") client_lib_dir("/usr/share/elasticsearch/lib") ); }; destination d_es_network { elasticsearch2( client-mode("http") index("network_${YEAR}.${MONTH}.${DAY}") type("syslog") # Description: The type of the index. For example, type("test") template("$(format-json --scope rfc3164 --scope nv-pairs --exclude R_DATE --key ISODATE)\n") cluster-url("http://elasticsearch:9200/") concurrent-requests("5") # Number of concurrrent batches flush_limit("5000") # The number of messages in a single batch skip-cluster-health-check("yes") cluster("clustername") client_lib_dir("/usr/share/elasticsearch/lib") ); };
Hi Scot, On Wed, Sep 13, 2017 at 04:30:26PM -0400, Scot wrote:
Has anyone had success/failure using multiple ES destinations in syslog-ng. I am want to direct traffic to different indexes based on syslog-ng filters
It is possible, but in your case not necessary: use a macro in the index name! Here's an example: destination d_es { elasticsearch2( ... index("${__es_index:-syslog}-${YEAR}.${MONTH}.${DAY}") ... template("$(format-json ... -x __* ...)") ... ); }; The template variable "${__es_index}" is set as usual using filters, channels and rewrite rules: filter f_syslog { ... }; filter f_network { ... }; rewrite r_syslog { set( "syslog", value("__es_index") ); }; rewrite r_network { set( "network", value("__es_index") ); }; log { source(...); junction { channel { filter(f_syslog); rewrite(r_syslog); }; channel { filter(f_network); rewrite(r_network); }; channel { flags(fallback); } }; destination(d_es); }; If you prefer having multiple destinations, it also works, but make sure you also explicitly set the persist-name: destination d_es_1 { elasticsearch2( ... persist-name('es_1') ... ); }; destination d_es_2 { elasticsearch2( ... persist-name('es_2') ... ); }; Cheers
participants (2)
-
Fabien Wernli
-
Scot