<div dir="ltr"><div>I mistakenly sent this to Szemere, László only.</div><div class="gmail_quote"><div dir="ltr"><div><br></div><div>Hi,</div><div><br></div><div>it seems I am almost there. <br></div><div>if I run</div><div> <br></div><div>syslog-ng -Fdv <br></div><div><br></div><div>(in the foreground) it executes without issue. <br></div><div>If I try to run it by doing </div><div><br></div><div>systemctl start syslog-ng <br></div><div>or <br></div><div>systemctl restart syslog-ng</div><div><br> </div><div>It crashes immediately:</div><div> # systemctl start syslog-ng<br> # systemctl status syslog-ng<br></div><div><span class="gmail-">● syslog-ng.service - System Logging Service<br> Loaded: loaded (/usr/lib/systemd/system/<wbr>syslog-ng.service; enabled; vendor preset: disabled)<br></span> Active: failed (Result: exit-code) since Mon 2018-07-09 11:05:04 CEST; 6s ago<br> Process: 3991 ExecStart=/usr/sbin/syslog-ng -F $SYSLOG_NG_PARAMS (code=exited, status=1/FAILURE)<br> Process: 3987 ExecStartPre=/usr/sbin/syslog-<wbr>ng-service-prepare (code=exited, status=0/SUCCESS)<br> Main PID: 3991 (code=exited, status=1/FAILURE)<br><br>Jul 09 11:05:04 syslog-test systemd[1]: Starting System Logging Service...<br>Jul 09 11:05:04 syslog-test systemd[1]: Started System Logging Service.<br>Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Main process exited, code=exited, status=1/FAILURE<br>Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Unit entered failed state.<br>Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Failed with result 'exit-code'.<br><br></div><div>Other info:</div><div># syslog-ng --version<br>syslog-ng 3 (3.16.1)<br>Config version: 3.16<br>Installer-Version: 3.16.1<span class="gmail-"><br>Revision: <br>Module-Directory: /usr/lib64/syslog-ng<br>Module-Path: /usr/lib64/syslog-ng<br></span>Available-Modules: add-contextual-data,affile,<wbr>afmongodb,afprog,afsocket,<wbr>afstomp,afuser,appmodel,<wbr>basicfuncs,cef,confgen,<wbr>cryptofuncs,csvparser,date,<wbr>dbparser,disk-buffer,graphite,<wbr>hook-commands,json-plugin,<wbr>kvformat,linux-kmsg-format,<wbr>map-value-pairs,pseudofile,<wbr>sdjournal,snmptrapd-parser,<wbr>stardate,syslogformat,system-<wbr>source,tags-parser,tfgetent,<wbr>xml,mod-java<span class="gmail-"><br>Enable-Debug: off<br>Enable-GProf: off<br>Enable-Memtrace: off<br>Enable-IPv6: on<br>Enable-Spoof-Source: on<br>Enable-TCP-Wrapper: on<br></span>Enable-Linux-Caps: on<br>Enable-Systemd: on<br></div><div><br></div><div>OpenSuse 42.3</div><div><br></div><div>Configuration:</div><div><span class="gmail-">############### Globale Optionen ###############<br>@version:3.16.1<br></span><span class="gmail-">@module mod-java # Elasticsearch .jar-libraries are located in /opt/syslog-ng/lib/syslog-ng/<wbr>java-modules/<br></span><span class="gmail-">@include "scl.conf"<br><br>options {<br> chain_hostnames(off); # Standard<br> flush_lines(0); # Standard<br> perm(0640); # Standard<br> stats_freq(3600); # Standard<br> threaded(yes); # Standard <br> create-dirs(yes); # erlaubt Syslog-NG, falls noetig, neue Verzeichnisse anzulegen<br> dir-owner(root); # die neuen Verzeichnisse gehoeren root<br> dir-perm(0640); # entspricht -rw-r-----, Besitzer: Lesen, Schreiben; Gruppe: Lesen; Alle: Nichts<br> file-template(t_<wbr>myLoggingFormat); # legt Standart-Template fuer file Destinations fest<br></span><span class="gmail-"># threaded(yes); # Example config Elasticsearch2<br># use-uniqid(yes); # Example config Elasticsearch2<br>};<br><br></span><span class="gmail-">############### Sources - Quellen ###############<br># Fuer interne Nachrichten<br>source s_myInternalSource {<br> system(); # von Betriebssystem und aehnlichem<br> internal(); # Syslog-interne Nachrichten<br>};<br><br># Fuer Netzwerk-Nachrichten<br>source s_myNetworkSource {<br></span>};</div><div><div><div class="gmail-h5"><br>############### Templates - Vorlagen ###############<br># Aufbau des Nachrichteninhalts fuer Dateiziele<br>template t_myLoggingFormat {<br> template("$(padding ${FULLHOST} 15 ' ')|${ISODATE}|PRI:$(padding ${PRI} 3 ' ')|${MSGHDR} ${MSG}\n");<br>};<br><br># Legacy-Nachrichten werden anders geparst<br># Mit Standardtempalte wir die Originalnachricht vollständig in MSG eingefügt<br><br># Fuer Nachrichtenpfad<br># Dateiname ist Tag(Nummer innerhalb des Monats)-Kuerzel(Mon, Tue, Wen, Thu, Fri, Sat, Sun)<br># zum Beispiel "<a href="http://127.0.0.1/2018/Jan/17-Sat" target="_blank">127.0.0.1/2018/Jan/17-Sat</a>"<br>template t_destination {<br> template("${FULLHOST}/${YEAR}/<wbr>${YEAR}-${MONTH_ABBREV}-${DAY}<wbr>.log");<br>};<br><br>############### Filter ###############<br># Ein Filter je Quelle<br># Filter kann man so machen: <br># filter <filter-id><br># {"<macro-or-template>" operator "<value-or-macro-or-template>"<wbr>};<br># oder mit Functions<br>filter f_noDebug {<br> level(emerg..info); <wbr> # Schließt Debug-Nachrichten aus <br>};<br><br>filter f_networkfilter {<br>};<br><br>filter f_checkpoints {<br>};<br><br><br>############### Rewrite Rules ######################<br>rewrite r_checkpoint_remove_irrelevant {<br> subst(<br> '\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} - ',<br> '',<br> type("pcre"),<br> value("MESSAGE") flags("utf8" "store-matches")<br> );<br><br>};<br><br></div></div>############### Parser #############################<br>parser p_pattern_db {<br> db-parser(<br> file("/opt/syslog-ng/etc/<wbr>patterndb.xml")<span class="gmail-"><br> );<br>};<br><br>############### Ziele - Destinations ###############<br># Ein Ziel fuer jede IP-Adresse, Jahr, Monat, Tag, etc<br>destination d_myDestination {<br> file("/var/log/syslog-ng/$<wbr>FULLHOST/$YEAR/$YEAR-$MONTH-$<wbr>DAY.log"<br> create_dirs(yes)<br> );<br>};<br><br>destination d_testination {<br> file("/var/log/syslog-ng/test/<wbr>$FULLHOST/$YEAR/$YEAR-$MONTH-$<wbr>DAY.log"<br> create_dirs(yes)<br> );<br>};<br><br></span><span class="gmail-">destination d_elasticsearch {<br> elasticsearch2(<br> cluster("syslog-ng")<br></span> cluster-url("<a href="http://127.0.0.1:9200" target="_blank">http://127.0.0.1:<wbr>9200</a>")<br> client-mode("http")<br># https-auth-type(basic)<br># http-auth-type-basic-password(<wbr>)<br># http-auth-type-basic-username(<wbr>)<span class="gmail-"><br> index("syslog-ng")<br> type("test")<br></span># default template($(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE}))<span class="gmail-"><br># template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)")<br></span><span class="gmail-"> )<br>};<br>############### Log-Pfade ###############<br># <br>log {<br> source(s_myNetworkSource); <wbr> # Fuer TCP und UDP Nachrichten von allen Clients<br> source(s_myInternalSource); <wbr> # interne Nachrichten<br> filter(f_networkfilter); <wbr> <br></span><span class="gmail-"> filter(f_noDebug); <wbr> # alle außer debug<br> destination(d_myDestination); <wbr> # universal Ziel; siehe Template<br>};<br>#<br></span>log { ###### Checkpoints #######<span class="gmail-"><br> source(s_myNetworkSource);<br> filter(f_checkpoints); <wbr> # Nur Checkpoint-Non-Standard<br> filter(f_noDebug);<br></span> rewrite(r_checkpoint_remove_<wbr>irrelevant);<br> destination(d_myDestination);<br>};<br>#<br>log { ###### Elasticsearch ######<br> source(s_myNetworkSource);<br> source(s_myInternalSource);<br> parser(p_pattern_db);<br> destination(d_elasticsearch);<br> flags(flow-control);<span class="gmail-"><br>};<br><br>#<br>#log { # for testing purposes<br># source(s_myNetworkSource);<br># source { syslog( ip( 0.0.0.0) transport("udp") flags(syslog-protocol); }; <br># destination(d_testination);<br>#};<br><br>############################## ENDE ##############################</span></div><div><br></div><div>If ran in the foreground this config logs message to disk and the elastic cluster receives the messages too.<br>I would also happily read what the difference is between starting the service and starting syslog-ng directly.<br>It (syslog-ng) appears to not be starting the service itself as systemctl status syslog-ng indicates.<br><br>Thank you for all the help so far. I am very greatful and could not have come anywhere near this point without your help.<br><br><br></div><div><br></div><div>%%%%%%%%%%%%%%%%% NEW: %%%%%%%%%%%%%%%%%%%<br></div><div><br></div><div>I am also working on the patterndb configuration. Starting syslog-ng no longer yields any configuration errors but it still rejects the file: <br></div><div><br></div><div># syslog-ng<br>log4j:ERROR Could not read configuration file [null].<br>java.lang.NullPointerException<br> at java.io.FileInputStream.<init>(FileInputStream.java:130)<br> at java.io.FileInputStream.<init>(FileInputStream.java:93)<br> at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:372)<br> at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:403)<br> at org.syslog_ng.elasticsearch_v2.client.http.ESHttpClient.<clinit>(ESHttpClient.java:67)<br> at org.syslog_ng.elasticsearch_v2.client.ESClientFactory.getESClient(ESClientFactory.java:43)<br> at org.syslog_ng.elasticsearch_v2.ElasticSearchDestination.init(ElasticSearchDestination.java:57)<br> at org.syslog_ng.LogPipe.initProxy(LogPipe.java:64)<br>log4j:ERROR Ignoring configuration file [null].<br>[main] INFO io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [<a href="http://127.0.0.1:9200">http://127.0.0.1:9200</a>]<br>[main] INFO io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager<br>[main] INFO io.searchbox.client.JestClientFactory - Using default GSON instance<br>[main] INFO io.searchbox.client.JestClientFactory - Node Discovery disabled...<br>[main] INFO io.searchbox.client.JestClientFactory - Idle connection reaping disabled...<br></div><div><br></div><div>My full configuration is the following. I have excluded the examples because the feature realtively sensitive information.</div><div>If one is absolutely necessary I can provide it in modified form. <br></div><div>I fixed all the remarks the of syntax errors syslog-ng threw on startup until nom remained but it still fails to load the config as of the message above.<br></div><div><br></div><div><?xml version='1.0' encoding='UTF-8'?><br><patterndb version='3' pub_date='2018-07-10'><br> <ruleset name='juniper'><br> <description><br> This ruleset covers the Messages sent by Juniper - PulseSecure.<br> </description><br> <url><a href="https://kb.pulsesecure.net/articles/Pulse_Secure_Article/KB40143">https://kb.pulsesecure.net/articles/Pulse_Secure_Article/KB40143</a></url><br> <pattern>PulseSecure</pattern><br> <rules><br> <rule provider="Niklas Deffner" id="1" class="Firewall"><br> <patterns><br> <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.hostname:()@@QSTRING:exchange.name:[:]@ - WebRequest ok : Host: @ESTRING:exchange.hostname:,@ Request: /Microsoft-Server-ActiveSync?User=@ESTRING:client.hostname:&@DeviceId=@ESTRING:client.id:&@DeviceType=@ESTRING:client.type:&@Cmd=@STRING@@ANYSTRING@</pattern><br> </patterns><br> <examples><br> <example><br> </example><br> </examples><br> <values><br> </values><br> <tags><br> <tag>WebRequest ok</tag><br> </tags><br> </rule><br><br> <rule provider="Niklas Deffner" id="2" class="Firewall"><br> <patterns><br> <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.hostname:()@@QSTRING:exchange.name:[:]@ - WebRequest completed, POST to https://@ESTRING:exchange.hostname::@443//Microsoft-Server-ActiveSync?User=@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id:&@DeviceType=@ESTRING:client.type:&@Cmd=@STRING:client.command@ from @IPv4:exchange.ip@ result=@NUMBER:result@ sent=@NUMBER:sent@ received=@NUMBER:received@ in @NUMBER:time@ seconds@ANYSTRING@</pattern><br> </patterns><br> <examples><br> <example><br> </example><br> </examples><br> <values><br> </values><br> <tags><br> <tag>WebRequest completed</tag><br> </tags><br> </rule><br><br> <rule provider="Niklas Deffner" id="3" class="Firewall"><br> <patterns><br> <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.name:()@@QSTRING:exchange.name:[:]@ - WebRequest ok : Host: @ESTRING:exchange.hostname:,@ Request: /Microsoft-Server-ActiveSync?Cmd=@ESTRING:client.command:&@User=@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id:&@DeviceType=@STRING:client.type@@ANYSTRING@</pattern><br> </patterns><br> <examples><br> <example><br> </example><br> </examples><br> <values><br> </values><br> <tags><br> <tag>WebRequest ok</tag><br> <tag> </tag><br> </tags><br> </rule><br><br> <rule provider="Niklas Deffner" id="4" class="Firewall"><br> <patterns><br> <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.name:()@@QSTRING:exchange.name:[:]@ - WebRequest completed, POST to https://@ESTRING:exchange.hostname::@443//Microsoft-Server-ActiveSync?Cmd=@ESTRING:client.command:&@User=@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id:&@DeviceType=@STRING:client.type@ from @IPv4:exchange.ip@ result=@NUMBER:result@ received=@NUMBER:received@ sent=@NUMBER:sent@ in @STRING:time@ @ANYSTRING@</pattern><br> </patterns><br> <examples><br> <example><br> </example><br> </examples><br> <values><br> </values><br> <tags><br> <tag>WebRequest completed</tag><br> <tag> </tag><br> </tags><br> </rule><br> </rules><br> </ruleset><br></patterndb><br></div><div><br></div><div><br></div><div>Sincerely,</div><div>Niklas Deffner<br></div><div><br></div></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">2018-07-06 8:42 GMT+02:00 T4iga <span dir="ltr"><<a href="mailto:niklastai97@gmail.com" target="_blank">niklastai97@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>Thank you for finding that bug and creating the pull request.</div><div>Will that be integrated into <a href="https://build.opensuse.org/project/show/home:czanik:syslog-ng316" target="_blank">https://build.opensuse.org/pro<wbr>ject/show/home:czanik:syslog-n<wbr>g316</a> too?<br></div><div>I have edited the aforementioned passage and removed the third argument and it is working again</div><div><br></div><div>Sincerely</div><div>Niklas Deffner<br></div><br></div><div class="gmail-m_4656447422826504720HOEnZb"><div class="gmail-m_4656447422826504720h5"><div class="gmail_extra"><br><div class="gmail_quote">2018-07-05 15:20 GMT+02:00 Szemere, László <span dir="ltr"><<a href="mailto:laszlo.szemere@balabit.com" target="_blank">laszlo.szemere@balabit.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello,<div><div> I successfully reproduced the crash with your configuration. The problem was in the <b>padding</b> template function, and has nothing to do with the earlier java problems.</div><div> If you intended to use the default (space) character for padding, you don't have to provide the third argument to the function. (Or provide the space character explicitly: <b>'' -> ' '</b>)</div><div><br></div><div> The problem exist, because in the padding function we divide with the length of the provided padding string <pre style="box-sizing:border-box;font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;font-size:12px;margin-top:0px;margin-bottom:0px;color:rgb(36,41,46);text-decoration-style:initial;text-decoration-color:initial;width:1px;height:1px"><a href="https://github.com/balabit/syslog-ng/blob/aa88bba2a9158fb6401bc260e0a58d7784c7bab4/modules/basicfuncs/str-funcs.c#L425" target="_blank">https://github.com/balabit/sys<wbr>log-ng/blob/aa88bba2a9158fb640<wbr>1bc260e0a58d7784c7bab4/modules<wbr>/basicfuncs/str-funcs.c#L425</a></pre></div><div><br></div><div> I will create a pull request to correct it.</div><div><br></div><div>Br,</div><div>Laci</div><div><br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-m_4656447422826504720m_335643469468912815h5">On Thu, Jul 5, 2018 at 11:16 AM, Fabien Wernli <span dir="ltr"><<a href="mailto:wernli@in2p3.fr" target="_blank">wernli@in2p3.fr</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-m_4656447422826504720m_335643469468912815h5"><span>On Thu, Jul 05, 2018 at 11:05:21AM +0200, T4iga wrote:<br>
> Hi Fabien Wernli,<br>
> <br>
> it says<br>
> sysctl kernel.core_pattern<br>
> kernel.core_pattern = |/usr/lib/systemd/systemd-core<wbr>dump %P %u %g %s %t %c<br>
> %e<br>
<br>
</span>Oh my...<br>
<br>
At least we now know who takes care of your core file!<br>
Reading<br>
<a href="https://www.freedesktop.org/software/systemd/man/systemd-coredump.html" rel="noreferrer" target="_blank">https://www.freedesktop.org/so<wbr>ftware/systemd/man/systemd-cor<wbr>edump.html</a> will<br>
probably help<br>
</div></div><div class="gmail-m_4656447422826504720m_335643469468912815m_2299002927954133661HOEnZb"><div class="gmail-m_4656447422826504720m_335643469468912815m_2299002927954133661h5"><br><span>
______________________________<wbr>______________________________<wbr>__________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" rel="noreferrer" target="_blank">https://lists.balabit.hu/mailm<wbr>an/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" rel="noreferrer" target="_blank">http://www.balabit.com/support<wbr>/documentation/?product=syslog<wbr>-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" rel="noreferrer" target="_blank">http://www.balabit.com/wiki/sy<wbr>slog-ng-faq</a><br>
<br>
</span></div></div></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></div><br></div>