Hi,
it seems I am almost there.
if I run
syslog-ng -Fdv
(in the foreground) it executes without issue.
If I try to run it by doing
systemctl start syslog-ng
or
systemctl restart syslog-ng
It crashes immediately:
# systemctl start syslog-ng
# systemctl status syslog-ng
● syslog-ng.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2018-07-09 11:05:04 CEST; 6s ago
Process: 3991 ExecStart=/usr/sbin/syslog-ng -F $SYSLOG_NG_PARAMS (code=exited, status=1/FAILURE)
Process: 3987 ExecStartPre=/usr/sbin/syslog-ng-service-prepare (code=exited, status=0/SUCCESS)
Main PID: 3991 (code=exited, status=1/FAILURE)
Jul 09 11:05:04 syslog-test systemd[1]: Starting System Logging Service...
Jul 09 11:05:04 syslog-test systemd[1]: Started System Logging Service.
Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Main process exited, code=exited, status=1/FAILURE
Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Unit entered failed state.
Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Failed with result 'exit-code'.
Other info:
# syslog-ng --version
syslog-ng 3 (3.16.1)
Config version: 3.16
Installer-Version: 3.16.1
Revision:
Module-Directory: /usr/lib64/syslog-ng
Module-Path: /usr/lib64/syslog-ng
Available-Modules: add-contextual-data,affile,afmongodb,afprog,afsocket,afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,hook-commands,json-plugin,kvformat,linux-kmsg-format,map-value-pairs,pseudofile,sdjournal,snmptrapd-parser,stardate,syslogformat,system-source,tags-parser,tfgetent,xml,mod-java
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: on
Enable-Linux-Caps: on
Enable-Systemd: on
OpenSuse 42.3
Configuration:
############### Globale Optionen ###############
@version:3.16.1
@module mod-java # Elasticsearch .jar-libraries are located in /opt/syslog-ng/lib/syslog-ng/java-modules/
@include "scl.conf"
options {
chain_hostnames(off); # Standard
flush_lines(0); # Standard
perm(0640); # Standard
stats_freq(3600); # Standard
threaded(yes); # Standard
create-dirs(yes); # erlaubt Syslog-NG, falls noetig, neue Verzeichnisse anzulegen
dir-owner(root); # die neuen Verzeichnisse gehoeren root
dir-perm(0640); # entspricht -rw-r-----, Besitzer: Lesen, Schreiben; Gruppe: Lesen; Alle: Nichts
file-template(t_myLoggingFormat); # legt Standart-Template fuer file Destinations fest
# threaded(yes); # Example config Elasticsearch2
# use-uniqid(yes); # Example config Elasticsearch2
};
############### Sources - Quellen ###############
# Fuer interne Nachrichten
source s_myInternalSource {
system(); # von Betriebssystem und aehnlichem
internal(); # Syslog-interne Nachrichten
};
# Fuer Netzwerk-Nachrichten
source s_myNetworkSource {
};
############### Templates - Vorlagen ###############
# Aufbau des Nachrichteninhalts fuer Dateiziele
template t_myLoggingFormat {
template("$(padding ${FULLHOST} 15 ' ')|${ISODATE}|PRI:$(padding ${PRI} 3 ' ')|${MSGHDR} ${MSG}\n");
};
# Legacy-Nachrichten werden anders geparst
# Mit Standardtempalte wir die Originalnachricht vollständig in MSG eingefügt
# Fuer Nachrichtenpfad
# Dateiname ist Tag(Nummer innerhalb des Monats)-Kuerzel(Mon, Tue, Wen, Thu, Fri, Sat, Sun)
# zum Beispiel "
127.0.0.1/2018/Jan/17-Sat"
template t_destination {
template("${FULLHOST}/${YEAR}/
${YEAR}-${MONTH_ABBREV}-${DAY}.log");
};
############### Filter ###############
# Ein Filter je Quelle
# Filter kann man so machen:
# filter <filter-id>
# {"<macro-or-template>" operator "<value-or-macro-or-template>"};
# oder mit Functions
filter f_noDebug {
level(emerg..info); # Schließt Debug-Nachrichten aus
};
filter f_networkfilter {
};
filter f_checkpoints {
};
############### Rewrite Rules ######################
rewrite r_checkpoint_remove_irrelevant {
subst(
'\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} - ',
'',
type("pcre"),
value("MESSAGE") flags("utf8" "store-matches")
);
};
############### Parser #############################
parser p_pattern_db {
db-parser(
file("/opt/syslog-ng/etc/
patterndb.xml")
);
};
############### Ziele - Destinations ###############
# Ein Ziel fuer jede IP-Adresse, Jahr, Monat, Tag, etc
destination d_myDestination {
file("/var/log/syslog-ng/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log"
create_dirs(yes)
);
};
destination d_testination {
file("/var/log/syslog-ng/test/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log"
create_dirs(yes)
);
};
destination d_elasticsearch {
elasticsearch2(
cluster("syslog-ng")
cluster-url("http://127.0.0.1:9200")
client-mode("http")
# https-auth-type(basic)
# http-auth-type-basic-password()
# http-auth-type-basic-username()
index("syslog-ng")
type("test")
# default template($(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE}))
# template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")
)
};
############### Log-Pfade ###############
#
log {
source(s_myNetworkSource); # Fuer TCP und UDP Nachrichten von allen Clients
source(s_myInternalSource); # interne Nachrichten
filter(f_networkfilter);
filter(f_noDebug); # alle außer debug
destination(d_myDestination); # universal Ziel; siehe Template
};
#
log { ###### Checkpoints #######
source(s_myNetworkSource);
filter(f_checkpoints); # Nur Checkpoint-Non-Standard
filter(f_noDebug);
rewrite(r_checkpoint_remove_irrelevant);
destination(d_myDestination);
};
#
log { ###### Elasticsearch ######
source(s_myNetworkSource);
source(s_myInternalSource);
parser(p_pattern_db);
destination(d_elasticsearch);
flags(flow-control);
};
#
#log { # for testing purposes
# source(s_myNetworkSource);
# source { syslog( ip( 0.0.0.0) transport("udp") flags(syslog-protocol); };
# destination(d_testination);
#};
############################## ENDE ##############################
If ran in the foreground this config logs message to disk and the elastic cluster receives the messages too.
I would also happily read what the difference is between starting the service and starting syslog-ng directly.
It (syslog-ng) appears to not be starting the service itself as systemctl status syslog-ng indicates.
Thank you for all the help so far. I am very greatful and could not have come anywhere near this point without your help.
%%%%%%%%%%%%%%%%% NEW: %%%%%%%%%%%%%%%%%%%
I am also working on the patterndb configuration. Starting syslog-ng no longer yields any configuration errors but it still rejects the file:
# syslog-ng
log4j:ERROR Could not read configuration file [null].
java.lang.NullPointerException
at java.io.FileInputStream.<init>(FileInputStream.java:130)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:372)
at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:403)
at org.syslog_ng.elasticsearch_v2.client.http.ESHttpClient.<clinit>(ESHttpClient.java:67)
at org.syslog_ng.elasticsearch_v2.client.ESClientFactory.getESClient(ESClientFactory.java:43)
at org.syslog_ng.elasticsearch_v2.ElasticSearchDestination.init(ElasticSearchDestination.java:57)
at org.syslog_ng.LogPipe.initProxy(LogPipe.java:64)
log4j:ERROR Ignoring configuration file [null].
[main] INFO io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [
http://127.0.0.1:9200]
[main] INFO io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager
[main] INFO io.searchbox.client.JestClientFactory - Using default GSON instance
[main] INFO io.searchbox.client.JestClientFactory - Node Discovery disabled...
[main] INFO io.searchbox.client.JestClientFactory - Idle connection reaping disabled...
My full configuration is the following. I have excluded the examples because the feature realtively sensitive information.
If one is absolutely necessary I can provide it in modified form.
I fixed all the remarks the of syntax errors syslog-ng threw on startup until nom remained but it still fails to load the config as of the message above.
<?xml version='1.0' encoding='UTF-8'?>
<patterndb version='3' pub_date='2018-07-10'>
<ruleset name='juniper'>
<description>
This ruleset covers the Messages sent by Juniper - PulseSecure.
</description>
<url>
https://kb.pulsesecure.net/articles/Pulse_Secure_Article/KB40143</url>
<pattern>PulseSecure</pattern>
<rules>
<rule provider="Niklas Deffner" id="1" class="Firewall">
<patterns>
<pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.hostname:()@@QSTRING:exchange.name:[:]@ - WebRequest ok : Host: @ESTRING:exchange.hostname:,@ Request: /Microsoft-Server-ActiveSync?User=@ESTRING:client.hostname:&@DeviceId=@ESTRING:client.id:&@DeviceType=@ESTRING:client.type:&@Cmd=@STRING@@ANYSTRING@</pattern>
</patterns>
<examples>
<example>
</example>
</examples>
<values>
</values>
<tags>
<tag>WebRequest ok</tag>
</tags>
</rule>
<rule provider="Niklas Deffner" id="2" class="Firewall">
<patterns>
<pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.hostname:()@@QSTRING:exchange.name:[:]@ - WebRequest completed, POST to https://@ESTRING:exchange.hostname::@443//Microsoft-Server-ActiveSync?User=@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id:&@DeviceType=@ESTRING:client.type:&@Cmd=@STRING:client.command@ from @IPv4:exchange.ip@ result=@NUMBER:result@ sent=@NUMBER:sent@ received=@NUMBER:received@ in @NUMBER:time@ seconds@ANYSTRING@</pattern>
</patterns>
<examples>
<example>
</example>
</examples>
<values>
</values>
<tags>
<tag>WebRequest completed</tag>
</tags>
</rule>
<rule provider="Niklas Deffner" id="3" class="Firewall">
<patterns>
<pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.name:()@@QSTRING:exchange.name:[:]@ - WebRequest ok : Host: @ESTRING:exchange.hostname:,@ Request: /Microsoft-Server-ActiveSync?Cmd=@ESTRING:client.command:&@User=@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id:&@DeviceType=@STRING:client.type@@ANYSTRING@</pattern>
</patterns>
<examples>
<example>
</example>
</examples>
<values>
</values>
<tags>
<tag>WebRequest ok</tag>
<tag> </tag>
</tags>
</rule>
<rule provider="Niklas Deffner" id="4" class="Firewall">
<patterns>
<pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.name:()@@QSTRING:exchange.name:[:]@ - WebRequest completed, POST to https://@ESTRING:exchange.hostname::@443//Microsoft-Server-ActiveSync?Cmd=@ESTRING:client.command:&@User=@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id:&@DeviceType=@STRING:client.type@ from @IPv4:exchange.ip@ result=@NUMBER:result@ received=@NUMBER:received@ sent=@NUMBER:sent@ in @STRING:time@ @ANYSTRING@</pattern>
</patterns>
<examples>
<example>
</example>
</examples>
<values>
</values>
<tags>
<tag>WebRequest completed</tag>
<tag> </tag>
</tags>
</rule>
</rules>
</ruleset>
</patterndb>
Sincerely,
Niklas Deffner