Trouble configuring elasticsearch2 destination
Hello Syslog-NG community, I am trying to configure syslog-ng to send messages to Elasticsearch for me to process them in Kibana. I get an error for the @module mod-java and the elasticsearch2 destination. I am running OpenSuse 42.3. syslog-ng --version: syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Available-Modules: add-contextual-data,afamqp,affile,afmongodb,afprog,afsocket,afstomp,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,sdjournal,syslogformat,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off I have downloaded and extracted Elasticsearch 6.3.0 and placed it in /usr/local/bin/elasticsearch-6.3.0/. In accordance to this: "Extract the Elasticsearch libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/elasticsearch/lib/ ) where syslog-ng OSE can access them. You must specify this directory in the syslog-ng OSE configuration file. The files are located in the lib directory and its subdirectories of the Elasticsearch release package." (quoted from Syslog-NG OSE 3.15 Admin Guide, 7.3.1. Procedure – Prerequisites, Step 3, page 175) I copied all JAR libraries inside elasticsearch-6.3.0/lib/ to the default path for syslog-ng plug-ins which is /usr/lib64/syslog-ng as stated below. Is this not what the step tells me to do? I get the following error: #[2018-07-03T11:20:39.403329] Plugin module not found in 'module-path'; module-path='/usr/lib64/syslog-ng', module='mod-java' Error parsing destination, destination plugin elasticsearch2 not found in /etc/syslog-ng/syslog-ng.conf at line 141, column 2: elasticsearch2( ^^^^^^^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng I seemt o be missing to necessary plug-ins: mod-java elasticsearch2 Where can I get these? Feel free to request any additional info I might have missed out on. Sincerely Niklas Deffner
Hi, The official distro packages do not contain the elasticsearch destinations (missing dependencies from the distros to build it, not just openSUSE but all others as well). There are unofficial packages with java/elasticsearch. You can find the latest version of unofficial syslog-ng packages for openSUSE here: https://build.opensuse.org/project/show/home:czanik:syslog-ng316 As I can see, you use ElasticSearch 6.3. In that case only the http(s) client mode is supported. Everything for that is included in the syslog-ng package, you do not need to copy .jar files around. Note, that libjvm.so needs to be available ( https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edit... ) either through LD_LIBRARY_PATH or by adding it to ld.so.conf (the later is recommended, unless you have multiple Java versions on your machine). See https://www.syslog-ng.com/community/b/blog/posts/troubleshooting-java-suppor... for more details. Let me know if you need any further help, Peter Peter Czanik (CzP) <peter.czanik@balabit.com> Balabit / syslog-ng upstream https://syslog-ng.com/blog/author/peterczanik/ https://twitter.com/PCzanik On Tue, Jul 3, 2018 at 11:34 AM, T4iga <niklastai97@gmail.com> wrote:
Hello Syslog-NG community,
I am trying to configure syslog-ng to send messages to Elasticsearch for me to process them in Kibana. I get an error for the @module mod-java and the elasticsearch2 destination.
I am running OpenSuse 42.3.
syslog-ng --version: syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Available-Modules: add-contextual-data,afamqp,affile,afmongodb,afprog, afsocket,afstomp,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,date, dbparser,disk-buffer,graphite,json-plugin,kvformat,linux- kmsg-format,pseudofile,sdjournal,syslogformat,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off
I have downloaded and extracted Elasticsearch 6.3.0 and placed it in /usr/local/bin/elasticsearch-6.3.0/. In accordance to this: "Extract the Elasticsearch libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/elasticsearch/lib/ ) where syslog-ng OSE can access them. You must specify this directory in the syslog-ng OSE configuration file. The files are located in the lib directory and its subdirectories of the Elasticsearch release package." (quoted from Syslog-NG OSE 3.15 Admin Guide, 7.3.1. Procedure – Prerequisites, Step 3, page 175) I copied all JAR libraries inside elasticsearch-6.3.0/lib/ to the default path for syslog-ng plug-ins which is /usr/lib64/syslog-ng as stated below. Is this not what the step tells me to do?
I get the following error:
#[2018-07-03T11:20:39.403329] Plugin module not found in 'module-path'; module-path='/usr/lib64/syslog-ng', module='mod-java' Error parsing destination, destination plugin elasticsearch2 not found in /etc/syslog-ng/syslog-ng.conf at line 141, column 2:
elasticsearch2( ^^^^^^^^^^^^^^
syslog-ng documentation: http://www.balabit.com/support/documentation/? product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng
I seemt o be missing to necessary plug-ins: mod-java elasticsearch2
Where can I get these? Feel free to request any additional info I might have missed out on.
Sincerely Niklas Deffner
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I managed to install syslog-ng-java-3.16 from your repo. I set the environment variable manually to my newest java implementation which is openjdk-1.8.0 and set up /etc/profile.local (as per https://unix.stackexchange.com/questions/117467/how-to-permanently-set-envir...) to supposedly do this every time I log in so I do not have to care about it. --syntax-only now no longer 'complains' however syslog does not successfully restart: syslog-test:/etc/syslog-ng # systemctl status -l syslog-ng ● syslog-ng.service - System Logging Service Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Die 2018-07-03 15:06:27 CEST; 12s ago Process: 13561 ExecStart=/usr/sbin/syslog-ng -F $SYSLOG_NG_PARAMS (code=exited, status=1/FAILURE) Process: 13556 ExecStartPre=/usr/sbin/syslog-ng-service-prepare (code=exited, status=0/SUCCESS) Main PID: 13561 (code=exited, status=1/FAILURE) Jul 03 15:06:27 syslog-test systemd[1]: Stopped System Logging Service. Jul 03 15:06:27 syslog-test systemd[1]: Starting System Logging Service... Jul 03 15:06:27 syslog-test systemd[1]: Started System Logging Service. Jul 03 15:06:27 syslog-test systemd[1]: syslog-ng.service: Main process exited, code=exited, status=1/FAILURE Jul 03 15:06:27 syslog-test systemd[1]: syslog-ng.service: Unit entered failed state. Jul 03 15:06:27 syslog-test systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Syslog-NG Config ############### Globale Optionen ############### @version:3.16 @module mod-java # Elasticsearch .jar-libraries are located in /opt/syslog-ng/lib/syslog-ng/java-modules/ @include "scl.conf" options { chain_hostnames(off); # Standard flush_lines(0); # Standard perm(0640); # Standard stats_freq(3600); # Standard threaded(yes); # Standard create-dirs(yes); # erlaubt Syslog-NG, falls noetig, neue Verzeichnisse anzulegen dir-owner(root); # die neuen Verzeichnisse gehoeren root dir-perm(0640); # entspricht -rw-r-----, Besitzer: Lesen, Schreiben; Gruppe: Lesen; Alle: Nichts file-template(t_myLoggingFormat); # legt Standart-Template fuer file Destinations fest # threaded(yes); # Example config Elasticsearch2 # use-uniqid(yes); # Example config Elasticsearch2 }; ############### Sources - Quellen ############### # Fuer interne Nachrichten source s_myInternalSource { system(); # von Betriebssystem und aehnlichem internal(); # Syslog-interne Nachrichten }; # Fuer Netzwerk-Nachrichten source s_myNetworkSource { }; ############### Templates - Vorlagen ############### # Aufbau des Nachrichteninhalts fuer Dateiziele template t_myLoggingFormat { template("$(padding ${FULLHOST} 15 '')|${ISODATE}|PRI:$(padding ${PRI} 3 '')|${MSGHDR} ${MSG}\n"); }; # Legacy-Nachrichten werden anders geparst # Mit Standardtempalte wir die Originalnachricht vollständig in MSG eingefügt # Fuer Nachrichtenpfad # Dateiname ist Tag(Nummer innerhalb des Monats)-Kuerzel(Mon, Tue, Wen, Thu, Fri, Sat, Sun) # zum Beispiel "127.0.0.1/2018/2018-01-17.log" template t_destination { template("${FULLHOST}/${YEAR}/${YEAR}-${MONTH_ABBREV}-${DAY}.log"); }; ############### Filter ############### # Ein Filter je Quelle # Filter kann man so machen: # filter <filter-id> # {"<macro-or-template>" operator "<value-or-macro-or-template>"}; # oder mit Functions filter f_noDebug { level(emerg..info); # Schließt Debug-Nachrichten aus }; filter f_networkfilter { }; ############### Parser ############################# parser pattern_db { db-parser( file("/opt/syslog-ng/etc/patterndb.xml") ); }; ############### Ziele - Destinations ############### # Ein Ziel fuer jede IP-Adresse, Jahr, Monat, Tag, etc destination d_myDestination { file("/var/log/syslog-ng/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log" create_dirs(yes) ); }; destination d_testination { file("/var/log/syslog-ng/test/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log" create_dirs(yes) ); }; destination d_elasticsearch { elasticsearch2( cluster("syslog-ng") # client-mode("http") index("syslog-ng") type("test") # template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") ) }; ############### Log-Pfade ############### # log { source(s_myNetworkSource); # Fuer TCP und UDP Nachrichten von allen Clients source(s_myInternalSource); # interne Nachrichten filter(f_networkfilter); filter(f_noDebug); # alle außer debug destination(d_myDestination); # universal Ziel; siehe Template }; # log { source(s_myNetworkSource); source(s_myInternalSource); parser(f_networkfilter); destination(d_elasticsearch); flags(flow-control); }; ################# Ende ################# Sincerely Niklas Deffner
Hi, On Tue, Jul 03, 2018 at 03:34:15PM +0200, T4iga wrote:
I managed to install syslog-ng-java-3.16 from your repo. I set the environment variable manually to my newest java implementation which is openjdk-1.8.0 and set up /etc/profile.local (as per https://unix.stackexchange.com/questions/117467/how-to-permanently-set-envir...) to supposedly do this every time I log in so I do not have to care about it.
This file won't be used by systemd when running the service. You have to add the LD_LIBRARY_PATH either in ld.so.conf or in the systemd unit file.
Hi, I think ld.so.conf will not work for me. I think I have multiple Java versions installed. I tried to read up on how to implement this in a unit file. It seems extremely complicated to fully understand compared to what I need it for. Would creating a file and setting the environment variable there as explained in the first answer here suffice? https://unix.stackexchange.com/questions/117467/how-to-permanently-set-envir... Is this universal amongst distributions using systemd? Sincerely Niklas Deffner 2018-07-03 15:43 GMT+02:00 Fabien Wernli <wernli@in2p3.fr>:
Hi,
On Tue, Jul 03, 2018 at 03:34:15PM +0200, T4iga wrote:
I managed to install syslog-ng-java-3.16 from your repo. I set the environment variable manually to my newest java implementation which is openjdk-1.8.0 and set up /etc/profile.local (as per https://unix.stackexchange.com/questions/117467/how-to- permanently-set-environmental-variables) to supposedly do this every time I log in so I do not have to care about it.
This file won't be used by systemd when running the service. You have to add the LD_LIBRARY_PATH either in ld.so.conf or in the systemd unit file.
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Tue, Jul 03, 2018 at 04:02:32PM +0200, T4iga wrote:
Hi,
I think ld.so.conf will not work for me. I think I have multiple Java versions installed. I tried to read up on how to implement this in a unit file. It seems extremely complicated to fully understand compared to what I need it for.
Would creating a file and setting the environment variable there as explained in the first answer here suffice? https://unix.stackexchange.com/questions/117467/how-to-permanently-set-envir...
Is this universal amongst distributions using systemd?
I don't see any reference to systemd in the thread you pasted. Chances are you can use /etc/sysconfig/ or /etc/default. Just check if you've got the corresponding EnvironmentFile entry in your systemd unit file. Here's what I have on my system: $ systemctl cat syslog-ng # /usr/lib/systemd/system/syslog-ng.service [Unit] Description=System Logger Daemon Documentation=man:syslog-ng(8) [Service] Type=notify ExecStart=/usr/sbin/syslog-ng -F $SYSLOGNG_OPTS -p /var/run/syslogd.pid ExecReload=/bin/kill -HUP $MAINPID EnvironmentFile=-/etc/default/syslog-ng EnvironmentFile=-/etc/sysconfig/syslog-ng StandardOutput=journal StandardError=journal Restart=on-failure as you can see, either /etc/default/syslog-ng or /etc/sysconfig/syslog-ng can be used to add the LD environment variable
participants (3)
-
Czanik, Péter
-
Fabien Wernli
-
T4iga