[syslog-ng] Fwd: Trouble configuring elasticsearch2 destination

10 Jul 2018

      I mistakenly sent this to Szemere, László only.

Hi,

it seems I am almost there.
if I run

syslog-ng -Fdv

(in the foreground) it executes without issue.
If I try to run it by doing

systemctl start syslog-ng
or
systemctl restart syslog-ng

It crashes immediately:
 # systemctl start syslog-ng
 # systemctl status syslog-ng
● syslog-ng.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled;
vendor preset: disabled)
   Active: failed (Result: exit-code) since Mon 2018-07-09 11:05:04 CEST;
6s ago
  Process: 3991 ExecStart=/usr/sbin/syslog-ng -F $SYSLOG_NG_PARAMS
(code=exited, status=1/FAILURE)
  Process: 3987 ExecStartPre=/usr/sbin/syslog-ng-service-prepare
(code=exited, status=0/SUCCESS)
 Main PID: 3991 (code=exited, status=1/FAILURE)

Jul 09 11:05:04 syslog-test systemd[1]: Starting System Logging Service...
Jul 09 11:05:04 syslog-test systemd[1]: Started System Logging Service.
Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Main process
exited, code=exited, status=1/FAILURE
Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Unit entered
failed state.
Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Failed with
result 'exit-code'.

Other info:
# syslog-ng --version
syslog-ng 3 (3.16.1)
Config version: 3.16
Installer-Version: 3.16.1
Revision:
Module-Directory: /usr/lib64/syslog-ng
Module-Path: /usr/lib64/syslog-ng
Available-Modules: add-contextual-data,affile,afmongodb,afprog,afsocket,
afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,
dbparser,disk-buffer,graphite,hook-commands,json-plugin,
kvformat,linux-kmsg-format,map-value-pairs,pseudofile,
sdjournal,snmptrapd-parser,stardate,syslogformat,system-
source,tags-parser,tfgetent,xml,mod-java
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on

OpenSuse 42.3

Configuration:
############### Globale Optionen ###############
@version:3.16.1
@module mod-java               # Elasticsearch .jar-libraries are located
in /opt/syslog-ng/lib/syslog-ng/java-modules/
@include "scl.conf"

options {
        chain_hostnames(off);   # Standard
        flush_lines(0);         # Standard
        perm(0640);             # Standard
        stats_freq(3600);       # Standard
        threaded(yes);          # Standard
        create-dirs(yes);       # erlaubt Syslog-NG, falls noetig, neue
Verzeichnisse anzulegen
        dir-owner(root);        # die neuen Verzeichnisse gehoeren root
        dir-perm(0640);         # entspricht -rw-r-----, Besitzer: Lesen,
Schreiben; Gruppe: Lesen; Alle: Nichts
        file-template(t_myLoggingFormat); # legt Standart-Template fuer
file Destinations fest
#       threaded(yes);          # Example config Elasticsearch2
#       use-uniqid(yes);        # Example config Elasticsearch2
};

############### Sources - Quellen ###############
# Fuer interne Nachrichten
source s_myInternalSource {
        system();       # von Betriebssystem und aehnlichem
        internal();     # Syslog-interne Nachrichten
};

# Fuer Netzwerk-Nachrichten
source s_myNetworkSource {
};

############### Templates - Vorlagen ###############
# Aufbau des Nachrichteninhalts fuer Dateiziele
template t_myLoggingFormat {
        template("$(padding ${FULLHOST} 15 ' ')|${ISODATE}|PRI:$(padding
${PRI} 3 ' ')|${MSGHDR} ${MSG}\n");
};

# Legacy-Nachrichten werden anders geparst
# Mit Standardtempalte wir die Originalnachricht vollständig in MSG
eingefügt

# Fuer Nachrichtenpfad
# Dateiname ist Tag(Nummer innerhalb des Monats)-Kuerzel(Mon, Tue, Wen,
Thu, Fri, Sat, Sun)
# zum Beispiel "127.0.0.1/2018/Jan/17-Sat"
template t_destination {
        template("${FULLHOST}/${YEAR}/${YEAR}-${MONTH_ABBREV}-${DAY}.log");
};

############### Filter ###############
# Ein Filter je Quelle
# Filter kann man so machen:
# filter <filter-id>
# {"<macro-or-template>" operator "<value-or-macro-or-template>"};
# oder mit Functions
filter f_noDebug {
        level(emerg..info);             # Schließt Debug-Nachrichten aus
};

filter f_networkfilter {
};

filter f_checkpoints {
};

############### Rewrite Rules ######################
rewrite r_checkpoint_remove_irrelevant {
        subst(
                '\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} - ',
                '',
                type("pcre"),
                value("MESSAGE") flags("utf8" "store-matches")
        );

};

############### Parser #############################
parser p_pattern_db {
        db-parser(
                file("/opt/syslog-ng/etc/patterndb.xml")
        );
};

############### Ziele - Destinations ###############
# Ein Ziel fuer jede IP-Adresse, Jahr, Monat, Tag, etc
destination d_myDestination {
        file("/var/log/syslog-ng/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log"
                create_dirs(yes)
        );
};

destination d_testination {
        file("/var/log/syslog-ng/test/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log"
                create_dirs(yes)
        );
};

destination d_elasticsearch {
        elasticsearch2(
                cluster("syslog-ng")
                cluster-url("http://127.0.0.1:9200")
                client-mode("http")
#               https-auth-type(basic)
#               http-auth-type-basic-password()
#               http-auth-type-basic-username()
                index("syslog-ng")
                type("test")
#       default template($(format-json --scope rfc5424 --exclude DATE --key
ISODATE @timestamp=${ISODATE}))
#               template("$(format-json --scope rfc5424 --scope nv-pairs
--exclude DATE --key ISODATE)")
        )
};
############### Log-Pfade ###############
#
log {
        source(s_myNetworkSource);              # Fuer TCP und UDP
Nachrichten von allen Clients
        source(s_myInternalSource);             # interne Nachrichten
        filter(f_networkfilter);
        filter(f_noDebug);                      # alle außer debug
        destination(d_myDestination);           # universal Ziel; siehe
Template
};
#
log {   ###### Checkpoints #######
        source(s_myNetworkSource);
        filter(f_checkpoints);          # Nur Checkpoint-Non-Standard
        filter(f_noDebug);
        rewrite(r_checkpoint_remove_irrelevant);
        destination(d_myDestination);
};
#
log {   ###### Elasticsearch ######
        source(s_myNetworkSource);
        source(s_myInternalSource);
        parser(p_pattern_db);
        destination(d_elasticsearch);
        flags(flow-control);
};

#
#log {  # for testing purposes
#       source(s_myNetworkSource);
#       source { syslog( ip( 0.0.0.0) transport("udp")
flags(syslog-protocol); };
#       destination(d_testination);
#};

############################## ENDE ##############################

If ran in the foreground this config logs message to disk and the elastic
cluster receives the messages too.
I would also happily read what the difference is between starting the
service and starting syslog-ng directly.
It (syslog-ng) appears to not be starting the service itself as systemctl
status syslog-ng indicates.

Thank you for all the help so far. I am very greatful and could not have
come anywhere near this point without your help.

%%%%%%%%%%%%%%%%% NEW: %%%%%%%%%%%%%%%%%%%

I am also working on the patterndb configuration. Starting syslog-ng no
longer yields any configuration errors but it still rejects the file:

# syslog-ng
log4j:ERROR Could not read configuration file [null].
java.lang.NullPointerException
    at java.io.FileInputStream.<init>(FileInputStream.java:130)
    at java.io.FileInputStream.<init>(FileInputStream.java:93)
    at
org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:372)
    at
org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:403)
    at
org.syslog_ng.elasticsearch_v2.client.http.ESHttpClient.<clinit>(ESHttpClient.java:67)
    at
org.syslog_ng.elasticsearch_v2.client.ESClientFactory.getESClient(ESClientFactory.java:43)
    at
org.syslog_ng.elasticsearch_v2.ElasticSearchDestination.init(ElasticSearchDestination.java:57)
    at org.syslog_ng.LogPipe.initProxy(LogPipe.java:64)
log4j:ERROR Ignoring configuration file [null].
[main] INFO io.searchbox.client.AbstractJestClient - Setting server pool to
a list of 1 servers: [http://127.0.0.1:9200]
[main] INFO io.searchbox.client.JestClientFactory - Using multi
thread/connection supporting pooling connection manager
[main] INFO io.searchbox.client.JestClientFactory - Using default GSON
instance
[main] INFO io.searchbox.client.JestClientFactory - Node Discovery
disabled...
[main] INFO io.searchbox.client.JestClientFactory - Idle connection reaping
disabled...

My full configuration is the following. I have excluded the examples
because the feature realtively sensitive information.
If one is absolutely necessary I can provide it in modified form.
I fixed all the remarks the of syntax errors syslog-ng threw on startup
until nom remained but it still fails to load the config as of the message
above.

<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='3' pub_date='2018-07-10'>
  <ruleset name='juniper'>
    <description>
      This ruleset covers the Messages sent by Juniper - PulseSecure.
    </description>
    <url>https://kb.pulsesecure.net/articles/Pulse_Secure_Article/KB40143
</url>
    <pattern>PulseSecure</pattern>
    <rules>
      <rule provider="Niklas Deffner" id="1" class="Firewall">
        <patterns>
          <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@]
@ESTRING:client.hostname:()@@QSTRING:exchange.name:[:]@ - WebRequest ok :
Host: @ESTRING:exchange.hostname:,@ Request:
/Microsoft-Server-ActiveSync?User=@ESTRING:client.hostname:&@DeviceId
=@ESTRING:client.id:&@DeviceType=@ESTRING:client.type:&@Cmd=@STRING@
@ANYSTRING@</pattern>
        </patterns>
        <examples>
          <example>
          </example>
        </examples>
        <values>
        </values>
        <tags>
          <tag>WebRequest ok</tag>
        </tags>
      </rule>

      <rule provider="Niklas Deffner" id="2" class="Firewall">
        <patterns>
          <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@]
@ESTRING:client.hostname:()@@QSTRING:exchange.name:[:]@ - WebRequest
completed, POST to https://@ESTRING:exchange.hostname::@443
//Microsoft-Server-ActiveSync?User=@ESTRING:client.hostname.url:&@DeviceId
=@ESTRING:client.id:&@DeviceType=@ESTRING:client.type:&@Cmd
=@STRING:client.command@ from @IPv4:exchange.ip@ result=@NUMBER:result@
sent=@NUMBER:sent@ received=@NUMBER:received@ in @NUMBER:time@
seconds@ANYSTRING@</pattern>
        </patterns>
        <examples>
          <example>
          </example>
        </examples>
        <values>
        </values>
        <tags>
          <tag>WebRequest completed</tag>
        </tags>
      </rule>

      <rule provider="Niklas Deffner" id="3" class="Firewall">
        <patterns>
          <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@]
@ESTRING:client.name:()@@QSTRING:exchange.name:[:]@ - WebRequest ok : Host:
@ESTRING:exchange.hostname:,@ Request:
/Microsoft-Server-ActiveSync?Cmd=@ESTRING:client.command:&@User
=@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id:
&@DeviceType=@STRING:client.type@@ANYSTRING@</pattern>
        </patterns>
        <examples>
          <example>
          </example>
        </examples>
        <values>
        </values>
        <tags>
          <tag>WebRequest ok</tag>
          <tag> </tag>
        </tags>
      </rule>

      <rule provider="Niklas Deffner" id="4" class="Firewall">
        <patterns>
          <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@]
@ESTRING:client.name:()@@QSTRING:exchange.name:[:]@ - WebRequest completed,
POST to https://@ESTRING:exchange.hostname::@443
//Microsoft-Server-ActiveSync?Cmd=@ESTRING:client.command:&@User
=@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id:
&@DeviceType=@STRING:client.type@ from @IPv4:exchange.ip@
result=@NUMBER:result@ received=@NUMBER:received@ sent=@NUMBER:sent@ in
@STRING:time@ @ANYSTRING@</pattern>
        </patterns>
        <examples>
          <example>
          </example>
        </examples>
        <values>
        </values>
        <tags>
          <tag>WebRequest completed</tag>
          <tag> </tag>
        </tags>
      </rule>
    </rules>
  </ruleset>
</patterndb>

Sincerely,
Niklas Deffner

2018-07-06 8:42 GMT+02:00 T4iga <niklastai97@gmail.com>:
...
Hi,
Thank you for finding that bug and creating the pull request.
Will that be integrated into https://build.opensuse.org/pro
ject/show/home:czanik:syslog-ng316 too?
I have edited the aforementioned passage and removed the third argument
and it is working again
Sincerely
Niklas Deffner
2018-07-05 15:20 GMT+02:00 Szemere, László <laszlo.szemere@balabit.com>:
...
Hello,
 I successfully reproduced the crash with your configuration. The problem
was in the *padding* template function, and has nothing to do with the
earlier java problems.
 If you intended to use the default (space) character for padding, you
don't have to provide the third argument to the function. (Or provide the
space character explicitly: *'' -> ' '*)
The problem exist, because in the padding function we divide with the
length of the provided padding string
https://github.com/balabit/syslog-ng/blob/aa88bba2a9158fb6401bc260e0a58d7784...
I will create a pull request to correct it.
Br,
Laci
On Thu, Jul 5, 2018 at 11:16 AM, Fabien Wernli <wernli@in2p3.fr> wrote:
...
On Thu, Jul 05, 2018 at 11:05:21AM +0200, T4iga wrote:
...
Hi Fabien Wernli,
it says
sysctl kernel.core_pattern
kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s
%t %c
%e
Oh my...
At least we now know who takes care of your core file!
Reading
https://www.freedesktop.org/software/systemd/man/systemd-coredump.html
will
probably help
____________________________________________________________
__________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support
/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq