Re: [syslog-ng] Trouble configuring elasticsearch2 destination

3 Jul 2018

      I managed to install syslog-ng-java-3.16 from your repo.
I set the environment variable manually to my newest java implementation
which is openjdk-1.8.0 and set up /etc/profile.local (as per
https://unix.stackexchange.com/questions/117467/how-to-permanently-set-envir...)
to supposedly do this every time I log in so I do not have to care about it.

--syntax-only now no longer 'complains' however syslog does not
successfully restart:

syslog-test:/etc/syslog-ng # systemctl status -l syslog-ng
● syslog-ng.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled;
vendor preset: disabled)
   Active: failed (Result: exit-code) since Die 2018-07-03 15:06:27 CEST;
12s ago
  Process: 13561 ExecStart=/usr/sbin/syslog-ng -F $SYSLOG_NG_PARAMS
(code=exited, status=1/FAILURE)
  Process: 13556 ExecStartPre=/usr/sbin/syslog-ng-service-prepare
(code=exited, status=0/SUCCESS)
 Main PID: 13561 (code=exited, status=1/FAILURE)

Jul 03 15:06:27 syslog-test systemd[1]: Stopped System Logging Service.
Jul 03 15:06:27 syslog-test systemd[1]: Starting System Logging Service...
Jul 03 15:06:27 syslog-test systemd[1]: Started System Logging Service.
Jul 03 15:06:27 syslog-test systemd[1]: syslog-ng.service: Main process
exited, code=exited, status=1/FAILURE
Jul 03 15:06:27 syslog-test systemd[1]: syslog-ng.service: Unit entered
failed state.
Jul 03 15:06:27 syslog-test systemd[1]: syslog-ng.service: Failed with
result 'exit-code'.

Syslog-NG Config
############### Globale Optionen ###############
@version:3.16
@module mod-java               # Elasticsearch .jar-libraries are located
in /opt/syslog-ng/lib/syslog-ng/java-modules/
@include "scl.conf"

options {
        chain_hostnames(off);   # Standard
        flush_lines(0);         # Standard
        perm(0640);             # Standard
        stats_freq(3600);       # Standard
        threaded(yes);          # Standard
        create-dirs(yes);       # erlaubt Syslog-NG, falls noetig, neue
Verzeichnisse anzulegen
        dir-owner(root);        # die neuen Verzeichnisse gehoeren root
        dir-perm(0640);         # entspricht -rw-r-----, Besitzer: Lesen,
Schreiben; Gruppe: Lesen; Alle: Nichts
        file-template(t_myLoggingFormat); # legt Standart-Template fuer
file Destinations fest
#       threaded(yes);          # Example config Elasticsearch2
#       use-uniqid(yes);        # Example config Elasticsearch2
};

############### Sources - Quellen ###############
# Fuer interne Nachrichten
source s_myInternalSource {
        system();       # von Betriebssystem und aehnlichem
        internal();     # Syslog-interne Nachrichten
};

# Fuer Netzwerk-Nachrichten
source s_myNetworkSource {
};

############### Templates - Vorlagen ###############
# Aufbau des Nachrichteninhalts fuer Dateiziele
template t_myLoggingFormat {
        template("$(padding ${FULLHOST} 15 '')|${ISODATE}|PRI:$(padding
${PRI} 3 '')|${MSGHDR} ${MSG}\n");
};

# Legacy-Nachrichten werden anders geparst
# Mit Standardtempalte wir die Originalnachricht vollständig in MSG
eingefügt

# Fuer Nachrichtenpfad
# Dateiname ist Tag(Nummer innerhalb des Monats)-Kuerzel(Mon, Tue, Wen,
Thu, Fri, Sat, Sun)
# zum Beispiel "127.0.0.1/2018/2018-01-17.log"
template t_destination {
        template("${FULLHOST}/${YEAR}/${YEAR}-${MONTH_ABBREV}-${DAY}.log");
};

############### Filter ###############
# Ein Filter je Quelle
# Filter kann man so machen:
# filter <filter-id>
# {"<macro-or-template>" operator "<value-or-macro-or-template>"};
# oder mit Functions
filter f_noDebug {
        level(emerg..info);             # Schließt Debug-Nachrichten aus
};

filter f_networkfilter {
};

############### Parser #############################
parser pattern_db {
        db-parser(
                file("/opt/syslog-ng/etc/patterndb.xml")
        );
};

############### Ziele - Destinations ###############
# Ein Ziel fuer jede IP-Adresse, Jahr, Monat, Tag, etc
destination d_myDestination {
        file("/var/log/syslog-ng/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log"
                create_dirs(yes)
        );
};

destination d_testination {
        file("/var/log/syslog-ng/test/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log"
                create_dirs(yes)
        );
};

destination d_elasticsearch {
        elasticsearch2(
                cluster("syslog-ng")
#               client-mode("http")
                index("syslog-ng")
                type("test")
#               template("$(format-json --scope rfc5424 --scope nv-pairs
--exclude DATE --key ISODATE)")
        )
};
############### Log-Pfade ###############
#
log {
        source(s_myNetworkSource);              # Fuer TCP und UDP
Nachrichten von allen Clients
        source(s_myInternalSource);             # interne Nachrichten
        filter(f_networkfilter);
        filter(f_noDebug);                      # alle außer debug
        destination(d_myDestination);           # universal Ziel; siehe
Template
};
#
log {
        source(s_myNetworkSource);
        source(s_myInternalSource);
        parser(f_networkfilter);
        destination(d_elasticsearch);
        flags(flow-control);
};
################# Ende #################

Sincerely

Niklas Deffner

Re: [syslog-ng] Trouble configuring elasticsearch2 destination

T4iga