[syslog-ng] Cannot send Syslog-ng to Elasticsearch
Nik Ambrosch
nik at ambrosch.com
Tue Jul 9 16:29:18 UTC 2019
It's pretty much a drop-in replacement for the java destination, so you
don't really need to do anything special. Here's what i use (template
omitted), hopefully that helps -
destination d_weblog_elastic {
elasticsearch-http(
index("logs-${YEAR}${MONTH}")
type("test")
persist-name("logs")
url("http://127.0.0.1:9200/_bulk")
time-zone("UTC")
template("$(format-json ...)")
);
};
This article contains some good reference material, you'd adjust slightly
to accommodate your environment -
https://www.syslog-ng.com/community/b/blog/posts/syslog-ng-and-elasticsearch-7-getting-started-on-rhel-centos
On Tue, Jul 9, 2019 at 11:31 AM Allen Olivas <allen.olivas at infodefense.com>
wrote:
> Hey Peter,
>
> I'll look into using the elasticsearch-http() destination. Does the
> elasticsearch-http() destination go directly into syslog-ng.conf or do I
> need to make a new .conf file (like elastic-http.conf) and add it to the
> conf.d/ directory? OR does it go in syslog-ng.conf and also the
> usr/share/syslog-ng/include/scl/elasticsearch/plugin.conf file?
>
> Per your request (and I do hope this helps illuminate things) I'm
> uploading our config files for syslog-ng,
> syslog-ng/.conf.d/elasticsearch.conf, and the plugin.conf
>
> *Syslog-ng.conf: *
>
> @version: 3.20
> @module mod-java
> @include "scl.conf"
> @define allow-config-dups 1
>
> # Syslog-ng configuration file, compatible with default Debian syslogd
> # installation.
>
> # First, set some global options.
> options { chain_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no);
> owner("root"); group("adm"); perm(0640); stats_freq(0);
> bad_hostname("^gconfd$");
> };
>
> ########################
> # Sources
> ########################
> # This is the default behavior of sysklogd package
> # Logs may come from unix stream, but not from another machine.
> #
> source s_src {
> system();
> internal();
> };
>
> # If you wish to get logs from remote machine you should uncomment
> # this and comment the above source line.
> #
> source s_net {
> tcp(port(514));
> udp(port(514));
> syslog();
> };
>
> ######
> # patterndb parser
> parser pattern_db {
> db-parser(
> file("/opt/syslog-ng/etc/patterndb.xml")
> );
> };
>
> ########################
> # Destinations
> ########################
> # First some standard logfile
> #
> destination d_auth { file("/var/log/auth.log"); };
> destination d_cron { file("/var/log/cron.log"); };
> destination d_daemon { file("/var/log/daemon.log"); };
> destination d_kern { file("/var/log/kern.log"); };
> destination d_lpr { file("/var/log/lpr.log"); };
> destination d_mail { file("/var/log/mail.log"); };
> destination d_syslog { file("/var/log/syslog"); };
> destination d_user { file("/var/log/user.log"); };
> destination d_uucp { file("/var/log/uucp.log"); };
>
> # This files are the log come from the mail subsystem.
> #
> destination d_mailinfo { file("/var/log/mail.info"); };
> destination d_mailwarn { file("/var/log/mail.warn"); };
> destination d_mailerr { file("/var/log/mail.err"); };
>
> # Logging for INN news system
> #
> destination d_newscrit { file("/var/log/news/news.crit"); };
> destination d_newserr { file("/var/log/news/news.err"); };
> destination d_newsnotice { file("/var/log/news/news.notice"); };
>
> # Some 'catch-all' logfiles.
> #
> destination d_debug { file("/var/log/debug"); };
> destination d_error { file("/var/log/error"); };
> destination d_messages { file("/var/log/messages"); };
>
> # The root's console.
> #
> destination d_console { usertty("root"); };
>
> # Virtual console.
> #
> destination d_console_all { file(`tty10`); };
>
> # The named pipe /dev/xconsole is for the nsole' utility. To use it,
> # you must invoke nsole' with the -file' option:
> #
> # $ xconsole -file /dev/xconsole [...]
> #
> destination d_xconsole { pipe("/dev/xconsole"); };
>
> # Send the messages to an other host
> #
> #destination d_net { tcp("127.0.0.1" port(1000) log_fifo_size(1000)); };
>
> #####
> ### Elasticsearch Destination
> #
> destination d_elastic { tcp("127.0.0.1" port(9200) template("$(format-json
> --scope selected_macros --scope nv_pairs --exclude DATE --key
> ISODATE)\n")); };
>
> # Debian only
> destination d_ppp { file("/var/log/ppp.log"); };
>
> ########################
> # Filters
> ########################
> # Here's come the filter options. With this rules, we can set which
> # message go where.
>
> filter f_dbg { level(debug); };
> filter f_info { level(info); };
> filter f_notice { level(notice); };
> filter f_warn { level(warn); };
> filter f_err { level(err); };
> filter f_crit { level(crit .. emerg); };
>
> filter f_debug { level(debug) and not facility(auth, authpriv, news,
> mail); };
> filter f_error { level(err .. emerg) ; };
> filter f_messages { level(info,notice,warn) and
> not facility(auth,authpriv,cron,daemon,mail,news); };
>
> filter f_auth { facility(auth, authpriv) and not filter(f_debug); };
> filter f_cron { facility(cron) and not filter(f_debug); };
> filter f_daemon { facility(daemon) and not filter(f_debug); };
> filter f_kern { facility(kern) and not filter(f_debug); };
> filter f_lpr { facility(lpr) and not filter(f_debug); };
> filter f_local { facility(local0, local1, local3, local4, local5,
> local6, local7) and not filter(f_debug); };
> filter f_mail { facility(mail) and not filter(f_debug); };
> filter f_news { facility(news) and not filter(f_debug); };
> filter f_syslog3 { not facility(auth, authpriv, mail) and not
> filter(f_debug); };
> filter f_user { facility(user) and not filter(f_debug); };
> filter f_uucp { facility(uucp) and not filter(f_debug); };
>
> filter f_cnews { level(notice, err, crit) and facility(news); };
> filter f_cother { level(debug, info, notice, warn) or facility(daemon,
> mail); };
>
> filter f_ppp { facility(local2) and not filter(f_debug); };
> filter f_console { level(warn .. emerg); };
>
> ########################
> # Log paths
> ########################
> log { source(s_src); filter(f_auth); destination(d_auth); };
> log { source(s_src); filter(f_cron); destination(d_cron); };
> log { source(s_src); filter(f_daemon); destination(d_daemon); };
> log { source(s_src); filter(f_kern); destination(d_kern); };
> log { source(s_src); filter(f_lpr); destination(d_lpr); };
> log { source(s_src); filter(f_syslog3); destination(d_syslog);
> destination(d_elastic); };
> log { source(s_src); filter(f_user); destination(d_user); };
> log { source(s_src); filter(f_uucp); destination(d_uucp); };
>
> log { source(s_src); filter(f_mail); destination(d_mail); };
> #log { source(s_src); filter(f_mail); filter(f_info);
> destination(d_mailinfo); };
> #log { source(s_src); filter(f_mail); filter(f_warn);
> destination(d_mailwarn); };
> #log { source(s_src); filter(f_mail); filter(f_err);
> destination(d_mailerr); };
>
> log { source(s_src); filter(f_news); filter(f_crit);
> destination(d_newscrit); };
> log { source(s_src); filter(f_news); filter(f_err);
> destination(d_newserr); };
> log { source(s_src); filter(f_news); filter(f_notice);
> destination(d_newsnotice); };
> #log { source(s_src); filter(f_cnews); destination(d_console_all); };
> #log { source(s_src); filter(f_cother); destination(d_console_all); };
>
> #log { source(s_src); filter(f_ppp); destination(d_ppp); };
>
> log { source(s_src); filter(f_debug); destination(d_debug); };
> log { source(s_src); filter(f_error); destination(d_error); };
> log { source(s_src); filter(f_messages); destination(d_messages); };
>
> log { source(s_src); filter(f_console); destination(d_console_all);
> destination(d_xconsole); };
> log { source(s_src); filter(f_crit); destination(d_console); };
>
> # All messages send to a remote site
> #
> #log { source(s_src); destination(d_net); };
>
> ###
> # Include all config files in /etc/syslog-ng/conf.d/
> ###
> @include "/etc/syslog-ng/conf.d/*.conf"
>
> *Elasticsearch.conf*
>
> @include "scl/elasticsearch/plugin.conf"
>
> source s_net { udp(); }; # All interfaces
> source s_src {
> system();
> internal();
> };
>
> block destination d_elastic() {
> elasticsearch2(
> client-lib-dir("/usr/share/elasticsearch/lib/")
> cluster("searchguard-demo")
> index("syslog-${YEAR}.${MONTH}.${DAY}")
> type("syslog")
> client-mode("https")
> # cluster-url("https://127.0.0.1:9200/")
> );
> };
>
> log {
> source(s_net);
> destination(d_elastic);
> flags(flow-control);
> };
>
>
> *Plugin.conf*
>
> ## scl/elasticsearch/plugin.conf -- Elasticsearch destination for syslog-ng
> ##
> ## Copyright (c) 2014 BalaBit IT Ltd, Budapest, Hungary
> ## Copyright (c) 2014 Gergely Nagy <algernon at balabit.hu>
> ##
> ## This program is free software; you can redistribute it and/or modify it
> ## under the terms of the GNU General Public License version 2 as published
> ## by the Free Software Foundation, or (at your option) any later version.
> ##
> ## This program is distributed in the hope that it will be useful,
> ## but WITHOUT ANY WARRANTY; without even the implied warranty of
> ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> ## GNU General Public License for more details.
> ##
> ## You should have received a copy of the GNU General Public License
> ## along with this program; if not, write to the Free Software
> ## Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301
> USA
> ##
> ## As an additional exemption you are allowed to compile & link against the
> ## OpenSSL libraries as published by the OpenSSL project. See the file
> ## COPYING for details.
>
> block destination d_elastic() {
> elasticsearch2(
> client-lib-dir("/usr/share/elasticsearch/lib/")
> index("syslog-${YEAR}.${MONTH}.${DAY}")
> type("syslog")
> client-mode("https")
> cluster-name("searchguard-demo")
> # cluster-url("https://127.0.0.1:9200/")
> );
> };
>
> Please let me know if there's anything else I can provide to help better
> understand and resolve this issue.
>
> Thanks,
>
> Allen Olivas
>
> -----Original Message-----
> From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> On Behalf Of Peter
> Kokai (pkokai)
> Sent: Tuesday, July 9, 2019 12:03 AM
> To: syslog-ng at lists.balabit.hu
> Subject: Re: [syslog-ng] Cannot send Syslog-ng to Elasticsearch
>
> Hello,
>
> Regarding your elasticsearch issue:
> Depending on your version I would suggest you to try out the new C based
> elasticsearch destination (there is no need for java setup).
>
> The commit that introduced gives an example how to configure:
>
> commit 381ceb14e578553faaef3ea005146cb988a9f444
> Refs: {origin/pr/2509}, syslog-ng-3.18.1-374-g381ceb14e
> Author: Zoltan Pallagi <pzolee at balabit.com>
> AuthorDate: Mon Feb 4 16:14:21 2019 +0100
> Commit: Zoltan Pallagi <pzolee at balabit.com>
> CommitDate: Mon Feb 4 16:14:21 2019 +0100
>
> Added elasticsearch-http() destination
>
> This destination is based on the native http destination of syslog-ng
> and uses elasticsearch bulk api (
> https://www.elastic.co/guide/en/elasticsearch/reference/6.5/docs-bulk.html
> )
>
> Example:
> destination d_elasticsearch_http {
> elasticsearch-http(index("my_index")
> type("my_type")
> url("http://my_elastic_server:9200/_bulk"));
> };
>
> Issue#1: I/O error occurred:
> This issue should not be related to sending data to elasticsearch, as that
> seems like a network destination, which tries to send data to 9200 port,
> but the server cuts the connection (probably because malformed data).
> Curios enough the 92000 is a standard elasticsearch port.
>
> Would you please share your configuration ? and/or what this destination
> supposed to do ?
>
> --
> kokan
>
> ________________________________________
> From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Allen
> Olivas <allen.olivas at infodefense.com>
> Sent: 08 July 2019 23:22
> To: syslog-ng at lists.balabit.hu
> Subject: [syslog-ng] Cannot send Syslog-ng to Elasticsearch
>
> CAUTION: This email originated from outside of the organization. Do not
> follow guidance, click links, or open attachments unless you recognize the
> sender and know the content is safe.
>
> Hello,
>
> Recently I’ve tried following along with the Syslog-NG to Elasticsearch
> and Kibana blog posts and Admin Documentation for integrating Syslog-NG
> into Elasticsearch but I’m unable to integrate the two.
>
> I see in the .conf files the destination calls for creating and Index
> Pattern for Syslog-NG but when I curl the existing indices I do not see
> syslog-ng.
>
> Also, I’m now receiving two errors. The first I’m fairly certain we need
> to resolve but I’ve not been able to find adequate documentation on how to
> identify the issue let along resolve it, and the second I’m not sure if we
> actually need to fix.
>
> The two issues:
>
> Issue#1: I/O error occurred
> syslog-ng[26432]: Syslog connection established; fd='12',
> server='AF_INET(127.0.0.1:9200)', local='AF_INET(0.0.0.0:0)'
> syslog-ng[26432]: I/O error occurred while writing; fd='12', error='Broken
> pipe (32)'
> syslog-ng[26432]: Syslog connection broken; fd='12',
> server='AF_INET(127.0.0.1:9200)', time_reopen='60'
>
> Issue#2: Error opening plugin module; module='mod-java', error='libjvm.so:
> cannot open shared object file: No such file or directory'
>
> For issue 1 I’m not sure what to do or how to resolve it. For issue 2, I
> know for certain libjvm does exist, and I’ve mapped the LD_LIBRARY_PATH to
> the directory libjvm.so resides in.
>
> Ultimately, are these two issues preventing Syslog-NG from sending to
> Elasticsearch or are they just separate issues to tackle after I get things
> cleared up, and most importantly if they’re not related, how do I integrate
> Syslog-NG with Elasticsearch and Kibana. Documentation is not helpful and
> not concise.
>
> Thanks!
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20190709/ab2c53ff/attachment-0001.html>
More information about the syslog-ng
mailing list