<div dir="ltr"><div>It's pretty much a drop-in replacement for the java destination, so you don't really need to do anything special. Here's what i use (template omitted), hopefully that helps -<br></div><div><br></div><div>destination d_weblog_elastic {<br> elasticsearch-http(<br> index("logs-${YEAR}${MONTH}")<br> type("test")<br> persist-name("logs")<br> url("<a href="http://127.0.0.1:9200/_bulk">http://127.0.0.1:9200/_bulk</a>")<br> time-zone("UTC")<br> template("$(format-json ...)")<br> );<br>};</div><div><br></div><div>This article contains some good reference material, you'd adjust slightly to accommodate your environment -</div><div><br></div><div><a href="https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-elasticsearch-7-getting-started-on-rhel-centos">https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-elasticsearch-7-getting-started-on-rhel-centos</a></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 9, 2019 at 11:31 AM Allen Olivas <<a href="mailto:allen.olivas@infodefense.com">allen.olivas@infodefense.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<font size="2" face="Calibri"><span style="font-size:11pt">
<div>Hey Peter, </div>
<div> </div>
<div>I'll look into using the elasticsearch-http() destination. Does the elasticsearch-http() destination go directly into syslog-ng.conf or do I need to make a new .conf file (like elastic-http.conf) and add it to the conf.d/ directory? OR does it go in syslog-ng.conf
and also the usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf file? </div>
<div> </div>
<div>Per your request (and I do hope this helps illuminate things) I'm uploading our config files for syslog-ng, syslog-ng/.conf.d/elasticsearch.conf, and the plugin.conf</div>
<div> </div>
<div><b>Syslog-ng.conf: </b></div>
<div> </div>
<div>@version: 3.20</div>
<div>@module mod-java</div>
<div>@include "scl.conf"</div>
<div>@define allow-config-dups 1</div>
<div> </div>
<div># Syslog-ng configuration file, compatible with default Debian syslogd</div>
<div># installation.</div>
<div> </div>
<div># First, set some global options.</div>
<div>options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);</div>
<div> owner("root"); group("adm"); perm(0640); stats_freq(0);</div>
<div> bad_hostname("^gconfd$");</div>
<div>};</div>
<div> </div>
<div>########################</div>
<div># Sources</div>
<div>########################</div>
<div># This is the default behavior of sysklogd package</div>
<div># Logs may come from unix stream, but not from another machine.</div>
<div>#</div>
<div>source s_src {</div>
<div> system();</div>
<div> internal();</div>
<div>};</div>
<div> </div>
<div># If you wish to get logs from remote machine you should uncomment</div>
<div># this and comment the above source line.</div>
<div>#</div>
<div>source s_net { </div>
<div> tcp(port(514));</div>
<div> udp(port(514));</div>
<div> syslog();</div>
<div>};</div>
<div> </div>
<div>######</div>
<div># patterndb parser</div>
<div>parser pattern_db {</div>
<div> db-parser(</div>
<div> file("/opt/syslog-ng/etc/patterndb.xml")</div>
<div> );</div>
<div>};</div>
<div> </div>
<div>########################</div>
<div># Destinations</div>
<div>########################</div>
<div># First some standard logfile</div>
<div>#</div>
<div>destination d_auth { file("/var/log/auth.log"); };</div>
<div>destination d_cron { file("/var/log/cron.log"); };</div>
<div>destination d_daemon { file("/var/log/daemon.log"); };</div>
<div>destination d_kern { file("/var/log/kern.log"); };</div>
<div>destination d_lpr { file("/var/log/lpr.log"); };</div>
<div>destination d_mail { file("/var/log/mail.log"); };</div>
<div>destination d_syslog { file("/var/log/syslog"); };</div>
<div>destination d_user { file("/var/log/user.log"); };</div>
<div>destination d_uucp { file("/var/log/uucp.log"); };</div>
<div> </div>
<div># This files are the log come from the mail subsystem.</div>
<div>#</div>
<div>destination d_mailinfo { file("/var/log/<a href="http://mail.info" target="_blank">mail.info</a>"); };</div>
<div>destination d_mailwarn { file("/var/log/mail.warn"); };</div>
<div>destination d_mailerr { file("/var/log/mail.err"); };</div>
<div> </div>
<div># Logging for INN news system</div>
<div>#</div>
<div>destination d_newscrit { file("/var/log/news/news.crit"); };</div>
<div>destination d_newserr { file("/var/log/news/news.err"); };</div>
<div>destination d_newsnotice { file("/var/log/news/news.notice"); };</div>
<div> </div>
<div># Some 'catch-all' logfiles.</div>
<div>#</div>
<div>destination d_debug { file("/var/log/debug"); };</div>
<div>destination d_error { file("/var/log/error"); };</div>
<div>destination d_messages { file("/var/log/messages"); };</div>
<div> </div>
<div># The root's console.</div>
<div>#</div>
<div>destination d_console { usertty("root"); };</div>
<div> </div>
<div># Virtual console.</div>
<div>#</div>
<div>destination d_console_all { file(`tty10`); };</div>
<div> </div>
<div># The named pipe /dev/xconsole is for the nsole' utility. To use it,</div>
<div># you must invoke nsole' with the -file' option:</div>
<div>#</div>
<div># $ xconsole -file /dev/xconsole [...]</div>
<div>#</div>
<div>destination d_xconsole { pipe("/dev/xconsole"); };</div>
<div> </div>
<div># Send the messages to an other host</div>
<div>#</div>
<div>#destination d_net { tcp("127.0.0.1" port(1000) log_fifo_size(1000)); };</div>
<div> </div>
<div>#####</div>
<div>### Elasticsearch Destination</div>
<div>#</div>
<div>destination d_elastic { tcp("127.0.0.1" port(9200) template("$(format-json --scope selected_macros --scope nv_pairs --exclude DATE --key ISODATE)\n")); };</div>
<div> </div>
<div># Debian only</div>
<div>destination d_ppp { file("/var/log/ppp.log"); };</div>
<div> </div>
<div>########################</div>
<div># Filters</div>
<div>########################</div>
<div># Here's come the filter options. With this rules, we can set which </div>
<div># message go where.</div>
<div> </div>
<div>filter f_dbg { level(debug); };</div>
<div>filter f_info { level(info); };</div>
<div>filter f_notice { level(notice); };</div>
<div>filter f_warn { level(warn); };</div>
<div>filter f_err { level(err); };</div>
<div>filter f_crit { level(crit .. emerg); };</div>
<div> </div>
<div>filter f_debug { level(debug) and not facility(auth, authpriv, news, mail); };</div>
<div>filter f_error { level(err .. emerg) ; };</div>
<div>filter f_messages { level(info,notice,warn) and </div>
<div> not facility(auth,authpriv,cron,daemon,mail,news); };</div>
<div> </div>
<div>filter f_auth { facility(auth, authpriv) and not filter(f_debug); };</div>
<div>filter f_cron { facility(cron) and not filter(f_debug); };</div>
<div>filter f_daemon { facility(daemon) and not filter(f_debug); };</div>
<div>filter f_kern { facility(kern) and not filter(f_debug); };</div>
<div>filter f_lpr { facility(lpr) and not filter(f_debug); };</div>
<div>filter f_local { facility(local0, local1, local3, local4, local5,</div>
<div> local6, local7) and not filter(f_debug); };</div>
<div>filter f_mail { facility(mail) and not filter(f_debug); };</div>
<div>filter f_news { facility(news) and not filter(f_debug); };</div>
<div>filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); };</div>
<div>filter f_user { facility(user) and not filter(f_debug); };</div>
<div>filter f_uucp { facility(uucp) and not filter(f_debug); };</div>
<div> </div>
<div>filter f_cnews { level(notice, err, crit) and facility(news); };</div>
<div>filter f_cother { level(debug, info, notice, warn) or facility(daemon, mail); };</div>
<div> </div>
<div>filter f_ppp { facility(local2) and not filter(f_debug); };</div>
<div>filter f_console { level(warn .. emerg); };</div>
<div> </div>
<div>########################</div>
<div># Log paths</div>
<div>########################</div>
<div>log { source(s_src); filter(f_auth); destination(d_auth); };</div>
<div>log { source(s_src); filter(f_cron); destination(d_cron); };</div>
<div>log { source(s_src); filter(f_daemon); destination(d_daemon); };</div>
<div>log { source(s_src); filter(f_kern); destination(d_kern); };</div>
<div>log { source(s_src); filter(f_lpr); destination(d_lpr); };</div>
<div>log { source(s_src); filter(f_syslog3); destination(d_syslog); destination(d_elastic); };</div>
<div>log { source(s_src); filter(f_user); destination(d_user); };</div>
<div>log { source(s_src); filter(f_uucp); destination(d_uucp); };</div>
<div> </div>
<div>log { source(s_src); filter(f_mail); destination(d_mail); };</div>
<div>#log { source(s_src); filter(f_mail); filter(f_info); destination(d_mailinfo); };</div>
<div>#log { source(s_src); filter(f_mail); filter(f_warn); destination(d_mailwarn); };</div>
<div>#log { source(s_src); filter(f_mail); filter(f_err); destination(d_mailerr); };</div>
<div> </div>
<div>log { source(s_src); filter(f_news); filter(f_crit); destination(d_newscrit); };</div>
<div>log { source(s_src); filter(f_news); filter(f_err); destination(d_newserr); };</div>
<div>log { source(s_src); filter(f_news); filter(f_notice); destination(d_newsnotice); };</div>
<div>#log { source(s_src); filter(f_cnews); destination(d_console_all); };</div>
<div>#log { source(s_src); filter(f_cother); destination(d_console_all); };</div>
<div> </div>
<div>#log { source(s_src); filter(f_ppp); destination(d_ppp); };</div>
<div> </div>
<div>log { source(s_src); filter(f_debug); destination(d_debug); };</div>
<div>log { source(s_src); filter(f_error); destination(d_error); };</div>
<div>log { source(s_src); filter(f_messages); destination(d_messages); };</div>
<div> </div>
<div>log { source(s_src); filter(f_console); destination(d_console_all);</div>
<div> destination(d_xconsole); };</div>
<div>log { source(s_src); filter(f_crit); destination(d_console); };</div>
<div> </div>
<div># All messages send to a remote site</div>
<div>#</div>
<div>#log { source(s_src); destination(d_net); };</div>
<div> </div>
<div>###</div>
<div># Include all config files in /etc/syslog-ng/conf.d/</div>
<div>###</div>
<div>@include "/etc/syslog-ng/conf.d/*.conf"</div>
<div> </div>
<div><b>Elasticsearch.conf</b></div>
<div> </div>
<div>@include "scl/elasticsearch/plugin.conf"</div>
<div> </div>
<div>source s_net { udp(); }; # All interfaces</div>
<div>source s_src {</div>
<div> system();</div>
<div> internal();</div>
<div>};</div>
<div> </div>
<div>block destination d_elastic() {</div>
<div> elasticsearch2(</div>
<div> client-lib-dir("/usr/share/elasticsearch/lib/")</div>
<div> cluster("searchguard-demo")</div>
<div> index("syslog-${YEAR}.${MONTH}.${DAY}")</div>
<div> type("syslog")</div>
<div> client-mode("https")</div>
<div># cluster-url("<a href="https://127.0.0.1:9200/" target="_blank">https://127.0.0.1:9200/</a>")</div>
<div> );</div>
<div>};</div>
<div> </div>
<div>log {</div>
<div> source(s_net);</div>
<div> destination(d_elastic);</div>
<div> flags(flow-control);</div>
<div>};</div>
<div> </div>
<div> </div>
<div><b>Plugin.conf</b></div>
<div> </div>
<div>## scl/elasticsearch/plugin.conf -- Elasticsearch destination for syslog-ng</div>
<div>##</div>
<div>## Copyright (c) 2014 BalaBit IT Ltd, Budapest, Hungary</div>
<div>## Copyright (c) 2014 Gergely Nagy <<a href="mailto:algernon@balabit.hu" target="_blank">algernon@balabit.hu</a>></div>
<div>##</div>
<div>## This program is free software; you can redistribute it and/or modify it</div>
<div>## under the terms of the GNU General Public License version 2 as published</div>
<div>## by the Free Software Foundation, or (at your option) any later version.</div>
<div>##</div>
<div>## This program is distributed in the hope that it will be useful,</div>
<div>## but WITHOUT ANY WARRANTY; without even the implied warranty of</div>
<div>## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the</div>
<div>## GNU General Public License for more details.</div>
<div>##</div>
<div>## You should have received a copy of the GNU General Public License</div>
<div>## along with this program; if not, write to the Free Software</div>
<div>## Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA</div>
<div>##</div>
<div>## As an additional exemption you are allowed to compile & link against the</div>
<div>## OpenSSL libraries as published by the OpenSSL project. See the file</div>
<div>## COPYING for details.</div>
<div> </div>
<div>block destination d_elastic() {</div>
<div> elasticsearch2(</div>
<div> client-lib-dir("/usr/share/elasticsearch/lib/")</div>
<div> index("syslog-${YEAR}.${MONTH}.${DAY}")</div>
<div> type("syslog")</div>
<div> client-mode("https")</div>
<div> cluster-name("searchguard-demo")</div>
<div># cluster-url("<a href="https://127.0.0.1:9200/" target="_blank">https://127.0.0.1:9200/</a>")</div>
<div> );</div>
<div>};</div>
<div> </div>
<div>Please let me know if there's anything else I can provide to help better understand and resolve this issue. </div>
<div> </div>
<div>Thanks, </div>
<div> </div>
<div>Allen Olivas</div>
<div> </div>
<div>-----Original Message-----<br>
From: syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a>> On Behalf Of Peter Kokai (pkokai)<br>
Sent: Tuesday, July 9, 2019 12:03 AM<br>
To: <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><br>
Subject: Re: [syslog-ng] Cannot send Syslog-ng to Elasticsearch</div>
<div> </div>
<div>Hello,</div>
<div> </div>
<div>Regarding your elasticsearch issue:</div>
<div>Depending on your version I would suggest you to try out the new C based elasticsearch destination (there is no need for java setup).</div>
<div> </div>
<div>The commit that introduced gives an example how to configure:</div>
<div> </div>
<div>commit 381ceb14e578553faaef3ea005146cb988a9f444</div>
<div>Refs: {origin/pr/2509}, syslog-ng-3.18.1-374-g381ceb14e</div>
<div>Author: Zoltan Pallagi <<a href="mailto:pzolee@balabit.com" target="_blank">pzolee@balabit.com</a>></div>
<div>AuthorDate: Mon Feb 4 16:14:21 2019 +0100</div>
<div>Commit: Zoltan Pallagi <<a href="mailto:pzolee@balabit.com" target="_blank">pzolee@balabit.com</a>></div>
<div>CommitDate: Mon Feb 4 16:14:21 2019 +0100</div>
<div> </div>
<div> Added elasticsearch-http() destination</div>
<div> </div>
<div> This destination is based on the native http destination of syslog-ng</div>
<div> and uses elasticsearch bulk api (<a href="https://www.elastic.co/guide/en/elasticsearch/reference/6.5/docs-bulk.html" target="_blank">https://www.elastic.co/guide/en/elasticsearch/reference/6.5/docs-bulk.html</a>)</div>
<div> </div>
<div> Example:</div>
<div> destination d_elasticsearch_http {</div>
<div> elasticsearch-http(index("my_index")</div>
<div> type("my_type")</div>
<div> url("<a href="http://my_elastic_server:9200/_bulk" target="_blank">http://my_elastic_server:9200/_bulk</a>"));</div>
<div> };</div>
<div> </div>
<div>Issue#1: I/O error occurred:</div>
<div>This issue should not be related to sending data to elasticsearch, as that seems like a network destination, which tries to send data to 9200 port, but the server cuts the connection (probably because malformed data). Curios enough the 92000 is a standard
elasticsearch port.</div>
<div> </div>
<div>Would you please share your configuration ? and/or what this destination supposed to do ?</div>
<div> </div>
<div>--</div>
<div>kokan</div>
<div> </div>
<div>________________________________________</div>
<div>From: syslog-ng <<a href="mailto:syslog-ng-bounces@lists.balabit.hu" target="_blank">syslog-ng-bounces@lists.balabit.hu</a>> on behalf of Allen Olivas <<a href="mailto:allen.olivas@infodefense.com" target="_blank">allen.olivas@infodefense.com</a>></div>
<div>Sent: 08 July 2019 23:22</div>
<div>To: <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a></div>
<div>Subject: [syslog-ng] Cannot send Syslog-ng to Elasticsearch</div>
<div> </div>
<div>CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.</div>
<div> </div>
<div>Hello,</div>
<div> </div>
<div>Recently I’ve tried following along with the Syslog-NG to Elasticsearch and Kibana blog posts and Admin Documentation for integrating Syslog-NG into Elasticsearch but I’m unable to integrate the two.</div>
<div> </div>
<div>I see in the .conf files the destination calls for creating and Index Pattern for Syslog-NG but when I curl the existing indices I do not see syslog-ng.</div>
<div> </div>
<div>Also, I’m now receiving two errors. The first I’m fairly certain we need to resolve but I’ve not been able to find adequate documentation on how to identify the issue let along resolve it, and the second I’m not sure if we actually need to fix.</div>
<div> </div>
<div>The two issues:</div>
<div> </div>
<div>Issue#1: I/O error occurred</div>
<div>syslog-ng[26432]: Syslog connection established; fd='12', server='AF_INET(127.0.0.1:9200)', local='AF_INET(<a href="http://0.0.0.0:0" target="_blank">0.0.0.0:0</a>)'</div>
<div>syslog-ng[26432]: I/O error occurred while writing; fd='12', error='Broken pipe (32)'</div>
<div>syslog-ng[26432]: Syslog connection broken; fd='12', server='AF_INET(127.0.0.1:9200)', time_reopen='60'</div>
<div> </div>
<div>Issue#2: Error opening plugin module; module='mod-java', error='libjvm.so: cannot open shared object file: No such file or directory'</div>
<div> </div>
<div>For issue 1 I’m not sure what to do or how to resolve it. For issue 2, I know for certain libjvm does exist, and I’ve mapped the LD_LIBRARY_PATH to the directory libjvm.so resides in.</div>
<div> </div>
<div>Ultimately, are these two issues preventing Syslog-NG from sending to Elasticsearch or are they just separate issues to tackle after I get things cleared up, and most importantly if they’re not related, how do I integrate Syslog-NG with Elasticsearch and
Kibana. Documentation is not helpful and not concise.</div>
<div> </div>
<div>Thanks!</div>
<div>______________________________________________________________________________</div>
<div>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a></div>
<div>Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a></div>
<div>FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a></div>
<div> </div>
<div> </div>
</span></font>
</div>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div>