- syslog-ng - lists.balabit.hu

Help With filtering
by Rose, Ed 20 Jul '18

20 Jul '18

Hello All I have syslog-ng 3.14 and I am trying to filter DNS logs. I have this working in a very old 2.0.9 version of syslog-ng. But it does not seem to want to work in the 3.14 environment. Can anyone help me out as I am pretty new to administering syslog-ng. Thanks ed Below is my code: ##New BlueCat config that doesn't work filter f_dhcpd { host(".*ddi-edns.*" flags(ignore-case)); and program("dhcpd"); }; destination d_dhcpd { file("/var/log2/gns-dmz/bluecat/${HOST}/dhcpd/log_${HOST}_dhcpd" perm(0640) group(splunk)); }; log { source(s_net); filter(f_dhcpd); destination(d_dhcpd); }; filter f_named { host(".*ddi-edns.*" flags(ignore-case)); and program("named"); }; destination d_named { file("/var/log2/gns-dmz/bluecat/${HOST}/named/log_${HOST}_named" perm(0640) group(splunk)); }; log { source(s_net); filter(f_named); destination(d_named); }; filter f_other { level(info, notice, warn, err, crit) and host(".*ddi-edns.*" flags(ignore-case)); and not program("dhcpd") and not program("named"); }; destination d_other { file("/var/log2/gns-dmz/bluecat/${HOST}/messages/log_${HOST}_messages" perm(0640) group(splunk)); }; log { source(s_net); filter(f_other); destination(d_other); }; ##Old BlueCat config that works #filter f_hsi-ddi-edns-1 { host("hsi-ddi-edns-1"); }; #filter f_ies-ddi-edns-1 { host("ies-ddi-edns-1"); }; #filter f_wv-ddi-edns-1 { host("wv-ddi-edns-1"); }; #destination d_hsi-ddi-edns-1 { file("/var/log2/gns-dmz/bluecat/hsi-ddi-edns-1/log_${HOST}"); }; #destination d_ies-ddi-edns-1 { file("/var/log2/gns-dmz/bluecat/ies-ddi-edns-1/log_${HOST}"); }; #destination d_wv-ddi-edns-1 { file("/var/log2/gns-dmz/bluecat/wv-ddi-edns-1/log_${HOST}"); }; #log { source(s_net); filter(f_hsi-ddi-edns-1); destination(d_hsi-ddi-edns-1); }; #log { source(s_net); filter(f_ies-ddi-edns-1); destination(d_ies-ddi-edns-1); }; #log { source(s_net); filter(f_wv-ddi-edns-1); destination(d_wv-ddi-edns-1); };

2 1

Fwd: Trouble configuring elasticsearch2 destination
by T4iga 13 Jul '18

13 Jul '18

I mistakenly sent this to Szemere, László only. Hi, it seems I am almost there. if I run syslog-ng -Fdv (in the foreground) it executes without issue. If I try to run it by doing systemctl start syslog-ng or systemctl restart syslog-ng It crashes immediately: # systemctl start syslog-ng # systemctl status syslog-ng ● syslog-ng.service - System Logging Service Loaded: loaded (/usr/lib/systemd/system/syslog-ng.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Mon 2018-07-09 11:05:04 CEST; 6s ago Process: 3991 ExecStart=/usr/sbin/syslog-ng -F $SYSLOG_NG_PARAMS (code=exited, status=1/FAILURE) Process: 3987 ExecStartPre=/usr/sbin/syslog-ng-service-prepare (code=exited, status=0/SUCCESS) Main PID: 3991 (code=exited, status=1/FAILURE) Jul 09 11:05:04 syslog-test systemd[1]: Starting System Logging Service... Jul 09 11:05:04 syslog-test systemd[1]: Started System Logging Service. Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Main process exited, code=exited, status=1/FAILURE Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Unit entered failed state. Jul 09 11:05:04 syslog-test systemd[1]: syslog-ng.service: Failed with result 'exit-code'. Other info: # syslog-ng --version syslog-ng 3 (3.16.1) Config version: 3.16 Installer-Version: 3.16.1 Revision: Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Available-Modules: add-contextual-data,affile,afmongodb,afprog,afsocket, afstomp,afuser,appmodel,basicfuncs,cef,confgen,cryptofuncs,csvparser,date, dbparser,disk-buffer,graphite,hook-commands,json-plugin, kvformat,linux-kmsg-format,map-value-pairs,pseudofile, sdjournal,snmptrapd-parser,stardate,syslogformat,system- source,tags-parser,tfgetent,xml,mod-java Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Systemd: on OpenSuse 42.3 Configuration: ############### Globale Optionen ############### @version:3.16.1 @module mod-java # Elasticsearch .jar-libraries are located in /opt/syslog-ng/lib/syslog-ng/java-modules/ @include "scl.conf" options { chain_hostnames(off); # Standard flush_lines(0); # Standard perm(0640); # Standard stats_freq(3600); # Standard threaded(yes); # Standard create-dirs(yes); # erlaubt Syslog-NG, falls noetig, neue Verzeichnisse anzulegen dir-owner(root); # die neuen Verzeichnisse gehoeren root dir-perm(0640); # entspricht -rw-r-----, Besitzer: Lesen, Schreiben; Gruppe: Lesen; Alle: Nichts file-template(t_myLoggingFormat); # legt Standart-Template fuer file Destinations fest # threaded(yes); # Example config Elasticsearch2 # use-uniqid(yes); # Example config Elasticsearch2 }; ############### Sources - Quellen ############### # Fuer interne Nachrichten source s_myInternalSource { system(); # von Betriebssystem und aehnlichem internal(); # Syslog-interne Nachrichten }; # Fuer Netzwerk-Nachrichten source s_myNetworkSource { }; ############### Templates - Vorlagen ############### # Aufbau des Nachrichteninhalts fuer Dateiziele template t_myLoggingFormat { template("$(padding ${FULLHOST} 15 ' ')|${ISODATE}|PRI:$(padding ${PRI} 3 ' ')|${MSGHDR} ${MSG}\n"); }; # Legacy-Nachrichten werden anders geparst # Mit Standardtempalte wir die Originalnachricht vollständig in MSG eingefügt # Fuer Nachrichtenpfad # Dateiname ist Tag(Nummer innerhalb des Monats)-Kuerzel(Mon, Tue, Wen, Thu, Fri, Sat, Sun) # zum Beispiel "127.0.0.1/2018/Jan/17-Sat" template t_destination { template("${FULLHOST}/${YEAR}/${YEAR}-${MONTH_ABBREV}-${DAY}.log"); }; ############### Filter ############### # Ein Filter je Quelle # Filter kann man so machen: # filter <filter-id> # {"<macro-or-template>" operator "<value-or-macro-or-template>"}; # oder mit Functions filter f_noDebug { level(emerg..info); # Schließt Debug-Nachrichten aus }; filter f_networkfilter { }; filter f_checkpoints { }; ############### Rewrite Rules ###################### rewrite r_checkpoint_remove_irrelevant { subst( '\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} - ', '', type("pcre"), value("MESSAGE") flags("utf8" "store-matches") ); }; ############### Parser ############################# parser p_pattern_db { db-parser( file("/opt/syslog-ng/etc/patterndb.xml") ); }; ############### Ziele - Destinations ############### # Ein Ziel fuer jede IP-Adresse, Jahr, Monat, Tag, etc destination d_myDestination { file("/var/log/syslog-ng/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log" create_dirs(yes) ); }; destination d_testination { file("/var/log/syslog-ng/test/$FULLHOST/$YEAR/$YEAR-$MONTH-$DAY.log" create_dirs(yes) ); }; destination d_elasticsearch { elasticsearch2( cluster("syslog-ng") cluster-url("http://127.0.0.1:9200") client-mode("http") # https-auth-type(basic) # http-auth-type-basic-password() # http-auth-type-basic-username() index("syslog-ng") type("test") # default template($(format-json --scope rfc5424 --exclude DATE --key ISODATE @timestamp=${ISODATE})) # template("$(format-json --scope rfc5424 --scope nv-pairs --exclude DATE --key ISODATE)") ) }; ############### Log-Pfade ############### # log { source(s_myNetworkSource); # Fuer TCP und UDP Nachrichten von allen Clients source(s_myInternalSource); # interne Nachrichten filter(f_networkfilter); filter(f_noDebug); # alle außer debug destination(d_myDestination); # universal Ziel; siehe Template }; # log { ###### Checkpoints ####### source(s_myNetworkSource); filter(f_checkpoints); # Nur Checkpoint-Non-Standard filter(f_noDebug); rewrite(r_checkpoint_remove_irrelevant); destination(d_myDestination); }; # log { ###### Elasticsearch ###### source(s_myNetworkSource); source(s_myInternalSource); parser(p_pattern_db); destination(d_elasticsearch); flags(flow-control); }; # #log { # for testing purposes # source(s_myNetworkSource); # source { syslog( ip( 0.0.0.0) transport("udp") flags(syslog-protocol); }; # destination(d_testination); #}; ############################## ENDE ############################## If ran in the foreground this config logs message to disk and the elastic cluster receives the messages too. I would also happily read what the difference is between starting the service and starting syslog-ng directly. It (syslog-ng) appears to not be starting the service itself as systemctl status syslog-ng indicates. Thank you for all the help so far. I am very greatful and could not have come anywhere near this point without your help. %%%%%%%%%%%%%%%%% NEW: %%%%%%%%%%%%%%%%%%% I am also working on the patterndb configuration. Starting syslog-ng no longer yields any configuration errors but it still rejects the file: # syslog-ng log4j:ERROR Could not read configuration file [null]. java.lang.NullPointerException at java.io.FileInputStream.<init>(FileInputStream.java:130) at java.io.FileInputStream.<init>(FileInputStream.java:93) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:372) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:403) at org.syslog_ng.elasticsearch_v2.client.http.ESHttpClient.<clinit>(ESHttpClient.java:67) at org.syslog_ng.elasticsearch_v2.client.ESClientFactory.getESClient(ESClientFactory.java:43) at org.syslog_ng.elasticsearch_v2.ElasticSearchDestination.init(ElasticSearchDestination.java:57) at org.syslog_ng.LogPipe.initProxy(LogPipe.java:64) log4j:ERROR Ignoring configuration file [null]. [main] INFO io.searchbox.client.AbstractJestClient - Setting server pool to a list of 1 servers: [http://127.0.0.1:9200] [main] INFO io.searchbox.client.JestClientFactory - Using multi thread/connection supporting pooling connection manager [main] INFO io.searchbox.client.JestClientFactory - Using default GSON instance [main] INFO io.searchbox.client.JestClientFactory - Node Discovery disabled... [main] INFO io.searchbox.client.JestClientFactory - Idle connection reaping disabled... My full configuration is the following. I have excluded the examples because the feature realtively sensitive information. If one is absolutely necessary I can provide it in modified form. I fixed all the remarks the of syntax errors syslog-ng threw on startup until nom remained but it still fails to load the config as of the message above. <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2018-07-10'> <ruleset name='juniper'> <description> This ruleset covers the Messages sent by Juniper - PulseSecure. </description> <url>https://kb.pulsesecure.net/articles/Pulse_Secure_Article/KB40143 </url> <pattern>PulseSecure</pattern> <rules> <rule provider="Niklas Deffner" id="1" class="Firewall"> <patterns> <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.hostname:()@@QSTRING:exchange.name:[:]@ - WebRequest ok : Host: @ESTRING:exchange.hostname:,@ Request: /Microsoft-Server-ActiveSync?User=@ESTRING:client.hostname:&@DeviceId =@ESTRING:client.id:&@DeviceType=@ESTRING:client.type:&@Cmd=@STRING@ @ANYSTRING@</pattern> </patterns> <examples> <example> </example> </examples> <values> </values> <tags> <tag>WebRequest ok</tag> </tags> </rule> <rule provider="Niklas Deffner" id="2" class="Firewall"> <patterns> <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.hostname:()@@QSTRING:exchange.name:[:]@ - WebRequest completed, POST to https://@ESTRING:exchange.hostname::@443 //Microsoft-Server-ActiveSync?User=@ESTRING:client.hostname.url:&@DeviceId =@ESTRING:client.id:&@DeviceType=@ESTRING:client.type:&@Cmd =@STRING:client.command@ from @IPv4:exchange.ip@ result=@NUMBER:result@ sent=@NUMBER:sent@ received=@NUMBER:received@ in @NUMBER:time@ seconds@ANYSTRING@</pattern> </patterns> <examples> <example> </example> </examples> <values> </values> <tags> <tag>WebRequest completed</tag> </tags> </rule> <rule provider="Niklas Deffner" id="3" class="Firewall"> <patterns> <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.name:()@@QSTRING:exchange.name:[:]@ - WebRequest ok : Host: @ESTRING:exchange.hostname:,@ Request: /Microsoft-Server-ActiveSync?Cmd=@ESTRING:client.command:&@User =@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id: &@DeviceType=@STRING:client.type@@ANYSTRING@</pattern> </patterns> <examples> <example> </example> </examples> <values> </values> <tags> <tag>WebRequest ok</tag> <tag> </tag> </tags> </rule> <rule provider="Niklas Deffner" id="4" class="Firewall"> <patterns> <pattern>psa@NUMBER:fw.id@ - [@IPv4:client.ip@] @ESTRING:client.name:()@@QSTRING:exchange.name:[:]@ - WebRequest completed, POST to https://@ESTRING:exchange.hostname::@443 //Microsoft-Server-ActiveSync?Cmd=@ESTRING:client.command:&@User =@ESTRING:client.hostname.url:&@DeviceId=@ESTRING:client.id: &@DeviceType=@STRING:client.type@ from @IPv4:exchange.ip@ result=@NUMBER:result@ received=@NUMBER:received@ sent=@NUMBER:sent@ in @STRING:time@ @ANYSTRING@</pattern> </patterns> <examples> <example> </example> </examples> <values> </values> <tags> <tag>WebRequest completed</tag> <tag> </tag> </tags> </rule> </rules> </ruleset> </patterndb> Sincerely, Niklas Deffner 2018-07-06 8:42 GMT+02:00 T4iga <niklastai97(a)gmail.com>: > Hi, > > Thank you for finding that bug and creating the pull request. > Will that be integrated into https://build.opensuse.org/pro > ject/show/home:czanik:syslog-ng316 too? > I have edited the aforementioned passage and removed the third argument > and it is working again > > Sincerely > Niklas Deffner > > > 2018-07-05 15:20 GMT+02:00 Szemere, László <laszlo.szemere(a)balabit.com>: > >> Hello, >> I successfully reproduced the crash with your configuration. The problem >> was in the *padding* template function, and has nothing to do with the >> earlier java problems. >> If you intended to use the default (space) character for padding, you >> don't have to provide the third argument to the function. (Or provide the >> space character explicitly: *'' -> ' '*) >> >> The problem exist, because in the padding function we divide with the >> length of the provided padding string >> >> https://github.com/balabit/syslog-ng/blob/aa88bba2a9158fb6401bc260e0a58d778… >> >> >> I will create a pull request to correct it. >> >> Br, >> Laci >> >> >> >> On Thu, Jul 5, 2018 at 11:16 AM, Fabien Wernli <wernli(a)in2p3.fr> wrote: >> >>> On Thu, Jul 05, 2018 at 11:05:21AM +0200, T4iga wrote: >>> > Hi Fabien Wernli, >>> > >>> > it says >>> > sysctl kernel.core_pattern >>> > kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %P %u %g %s >>> %t %c >>> > %e >>> >>> Oh my... >>> >>> At least we now know who takes care of your core file! >>> Reading >>> https://www.freedesktop.org/software/systemd/man/systemd-coredump.html >>> will >>> probably help >>> >>> ____________________________________________________________ >>> __________________ >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng >>> Documentation: http://www.balabit.com/support >>> /documentation/?product=syslog-ng >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq >>> >>> >> >

3 6

RE syslog-ng OSE parsing
by Daniel Ehrlich 10 Jul '18

10 Jul '18

Hi, Hoping you can assist me, I haven't really come across anything that makes full sense to me in my searching of various sites/forums. This is an example log. Jun 14 11:57:27 PM.685 UTC : %UC_LOGIN-4-AuthenticationFailed: %[Login Date/Time=06/15/2018 at 09:57:27][Login IP Address/Hostname=10.25.1.16][Login Interface=cucm-uds][Login UserID=POBAR][App ID=Cisco Tomcat][Cluster ID=][Node ID=cucmsub-prd-t2]: Login Authentication failed. Syslog-ng reads the $HOST as PM.685 ; can I get it to rewrite host as cucmsub-prd-t2 ? i.e. Node ID= Thanks you Kind Regards, Daniel Ehrlich MastInfoSysSec, DipBA, SSCP, F5-CA, Splunk CA ICT Security Officer ICT Client Services|Infrastructure Services Phone: +61 7 4687 5600 Email: <mailto:Daniel.Ehrlich@usq.edu.au> Daniel.Ehrlich(a)usq.edu.au Toowoomba | Queensland | 4350 | Australia _____________________________________________________________ This email (including any attached files) is confidential and is for the intended recipient(s) only. If you received this email by mistake, please, as a courtesy, tell the sender, then delete this email. The views and opinions are the originator's and do not necessarily reflect those of the University of Southern Queensland. Although all reasonable precautions were taken to ensure that this email contained no viruses at the time it was sent we accept no liability for any losses arising from its receipt. The University of Southern Queensland is a registered provider of education with the Australian Government. (CRICOS Institution Code QLD 00244B / NSW 02225M, TEQSA PRV12081 )

3 8

Re: [syslog-ng] Trouble configuring elasticsearch2 destination
by Fabien Wernli 05 Jul '18

05 Jul '18

In order to put aside any systemd complexities, let's try to run syslog-ng in the foreground: export LD_LIBRARY_PATH=/usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/amd64/server syslog-ng -Fdv What happens then?

5 15

syslog-ng.com upgrade
by Gergely Bodnár 04 Jul '18

04 Jul '18

Dear syslog-ng users, As you are likely aware, Balabit is now part of One Identity, and on July 1, we reached a major milestone in our integration schedule: we successfully closed the migration for most of our business systems and applications. While many of these are back office related and thus invisible to our open source users and customers, the syslog-ng.com website also received a major infrastructure upgrade. While the product pages look roughly the same as the old ones, you will notice a big difference when navigating to the technical documentation. - A new look and feel, in accordance with the broader Quest support portal. - The older syslog-ng Open Source Edition documentation (prior 3.16) is available only in PDF format. - Going forward, only multi-page HTML and PDF versions will be available of the Administrator' Guide. Single page guides are no longer offered. - The Debian Jessie repository of Open Source Edition is currently down. The past couple months showed us that there is real interest from the community for official repositories, and in the future, we would like to expand on our previous offer with more operating systems supported. However, we currently do not have an exact timeline to share with you. As with all migrations of this scale, there ought to be bugs surfacing in the coming days and weeks. We kindly ask you to report them by replying to this email, and we will do our best to fix them as our time permits. I would also take the opportunity and reassure you that syslog-ng Open Source Edition is here to stay, with the level of enthusiasm from our developers and technical writers that has been the hallmark of syslog-ng OSE for more than a decade. Regards, Gergely Bodnar -- *Gergely Bodnar* | syslog-ng Product Owner gergely.bodnar(a)balabit.com follow Balabit on: web <http://go.balabit.com/F9ZZ500bh0U013LX000VAE5> | blog <http://go.balabit.com/VZZi000AL90V300U55X1cE0> | linkedin <http://go.balabit.com/CZ100j0d50LU0EAZ3X09V05>| facebook <http://go.balabit.com/I090eUZ0551kEA3X0L0ZV00> | twitter <http://go.balabit.com/c0503X1U05EZ09AVl0ZfL00> *© BalaBit Corp.* Weiser Mazars 135 West 50th Street New York, NY 10020 USA

3 4

Trouble configuring elasticsearch2 destination
by T4iga 03 Jul '18

03 Jul '18

Hello Syslog-NG community, I am trying to configure syslog-ng to send messages to Elasticsearch for me to process them in Kibana. I get an error for the @module mod-java and the elasticsearch2 destination. I am running OpenSuse 42.3. syslog-ng --version: syslog-ng 3.8.1 Installer-Version: 3.8.1 Revision: Module-Directory: /usr/lib64/syslog-ng Module-Path: /usr/lib64/syslog-ng Available-Modules: add-contextual-data,afamqp,affile,afmongodb,afprog,afsocket,afstomp,afuser,basicfuncs,cef,confgen,cryptofuncs,csvparser,date,dbparser,disk-buffer,graphite,json-plugin,kvformat,linux-kmsg-format,pseudofile,sdjournal,syslogformat,system-source Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: off I have downloaded and extracted Elasticsearch 6.3.0 and placed it in /usr/local/bin/elasticsearch-6.3.0/. In accordance to this: "Extract the Elasticsearch libraries into a temporary directory, then collect the various .jar files into a single directory (for example, /opt/elasticsearch/lib/ ) where syslog-ng OSE can access them. You must specify this directory in the syslog-ng OSE configuration file. The files are located in the lib directory and its subdirectories of the Elasticsearch release package." (quoted from Syslog-NG OSE 3.15 Admin Guide, 7.3.1. Procedure – Prerequisites, Step 3, page 175) I copied all JAR libraries inside elasticsearch-6.3.0/lib/ to the default path for syslog-ng plug-ins which is /usr/lib64/syslog-ng as stated below. Is this not what the step tells me to do? I get the following error: #[2018-07-03T11:20:39.403329] Plugin module not found in 'module-path'; module-path='/usr/lib64/syslog-ng', module='mod-java' Error parsing destination, destination plugin elasticsearch2 not found in /etc/syslog-ng/syslog-ng.conf at line 141, column 2: elasticsearch2( ^^^^^^^^^^^^^^ syslog-ng documentation: http://www.balabit.com/support/documentation/?product=syslog-ng mailing list: https://lists.balabit.hu/mailman/listinfo/syslog-ng I seemt o be missing to necessary plug-ins: mod-java elasticsearch2 Where can I get these? Feel free to request any additional info I might have missed out on. Sincerely Niklas Deffner

3 5

(U) Find the source systems that write to specific Syslog Server
by Amin, Jitesh CTR DISA JSP (US) 03 Jul '18

03 Jul '18

CLASSIFICATION: UNCLASSIFIED Hello, We have multiple servers running syslog. By looking at the syslong.conf file we can identify where the syslog servers are forwarding the data to. But what we really want to know is what all sources are writing their logs to our syslog servers. Is there a way for us to look somewhere within syslog configuration and find out which all systems are forwarding/writing logs to a specific syslog server? Thanks Jitesh AminCLASSIFICATION: UNCLASSIFIED

2 1

Re: [syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL
by Donatello D 02 Jul '18

02 Jul '18

@Jim - this is what i use as a workaround already, but it is sub-optimal, as there will be other files that are rolled over in different intervals, so i end up reloading config multiple times. the real question is why does it work in RHEL and fail in UBUNTU? > Date: Sat, 30 Jun 2018 20:31:59 -0400 > From: Jim Hendrick <james.r.hendrick(a)gmail.com> > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly on > UBUNTU, works fine in RHEL > Message-ID: > <CANEn2idABV25G1vFa4B=WhOyuHjd3HwLMKFBHgqydH6zvH0H9w(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > hardlinks are simply additional pointers to the same inode. > symlinks contain the name of the referenced object > > If you want a hack - maybe have syslog-ng reload after the file changes? > (SIGHUP) > > > > On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal <ankit(a)travelmyheart.org> > wrote: > > > Hi, > > > > I ran into a similar problem on Ubuntu as well. > > > > In my case, I was tracking the Tomcat localhost log file in syslog-ng but > > Tomcat creates a new log file everyday by default, and the filename changes > > (since it includes the date). > > > > Therefore, I periodically created a softlink to the localhost log file > > where the link had a constant name. The constant name is needed because I > > obviously cannot keep changing the syslog-ng configuration to match the > > day's localhost log file name. > > > > I found that the softlink did not work. > > > > Instead I had to create a hardlink. > > > > This is because the softlink's modified date does not change when the > > underlying file changes. The hardlink's modified date does change since it > > is pointing to the actual data. We need the modified date to change for the > > syslog-ng client to pick up new log entries. > > > > In my case, I periodically ran the following command via CRON in the > > Tomcat logs directory: > > > > sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log > > > > This is to get the latest localhost log file and create the hardlink for > > it (overwriting the older hardlink that may have been pointing to the > > previous day's localhost log file). > > > > I ran this every hour just to be safe. > > > > > > So in your case, I think you would just need to recreate the hardlink as > > soon as your log file is rotated. > > > > > > Hope this helps. > > > > Ankit > > > > > > > > ---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D > > <bluray.vik(a)gmail.com <bluray.vik(a)gmail.com>>* wrote ---- > > > > syslog-ng is configured to read a symlink pointing to logs generated from > > my application which rotates the file using log4j2 rollingfile appender. > > Everything works fine till the rotation happens. after the file get rotated > > syslog-ng still seems to hold on to the older inode (which is not moved) > > and doesn't change to follow the new logs. this however does not happen in > > RHEL where syslog-ng recognizes the file is now rotated and moves to the > > new file. In both cases the sym link is always configured to point to the > > latest file. version details and logs from both OSs below. > > > > What am i missing here? > > > > UBUNTU - > > syslog-ng 3.5.6 > > Installer-Version: 3.5.6 > > Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04) > > Compile-Date: Oct 24 2015 03:49:19 > > Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser, > > syslogformat,afamqp,redis,afsql,affile,afsmtp,linux- > > kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs, > > json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb > > Enable-Debug: off > > Enable-GProf: off > > Enable-Memtrace: off > > Enable-IPv6: on > > Enable-Spoof-Source: on > > Enable-TCP-Wrapper: on > > Enable-Linux-Caps: on > > Enable-Pcre: on > > > > symlink is pointing to the file that gets the logs. prior to rotation the > > process watches correctly for the file (same inodes held by my app and > > syslog-ng) > > > > lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log -> > > /x/logs/vik-test_access.log > > > > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > > java 11032 vikram 53w REG 8,1 1101 1542626 > > vik-test_access.log > > syslog-ng 21661 root 9r REG 8,1 1101 1542626 > > vik-test_access.log > > > > > > Post rotation, syslog-ng holds on to the older file (now rotated). > > > > COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > > java 11032 vikram 53w REG 8,1 876 1542631 > > e/elasticsearch-6.2.3/logs/vik-test_access.log > > syslog-ng 21661 root 9r REG 8,1 1101 1542626 > > e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log > > > > The same setup works perfectly fine in RHEL (version details below) where > > syslog-ng follows the new file correctly. > > > > RHEL > > syslog-ng 3.3.5 > > Installer-Version: 3.3.5 > > Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog- > > ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722 > > Compile-Date: Oct 18 2012 15:17:09 > > Default-Modules: affile,afprog,afsocket,afuser, > > basicfuncs,csvparser,dbparser,syslogformat > > Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs, > > basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser > > Enable-Debug: off > > Enable-GProf: off > > Enable-Memtrace: off > > Enable-IPv6: on > > Enable-Spoof-Source: off > > Enable-TCP-Wrapper: on > > Enable-Linux-Caps: off > > Enable-Pcre: on > > > > ____________________________________________________________ > > __________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/? > > product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > > > > > ____________________________________________________________ > > __________________ > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > Documentation: http://www.balabit.com/support/documentation/? > > product=syslog-ng > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > > > > > > >

4 6

syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL
by Donatello D 01 Jul '18

01 Jul '18

syslog-ng is configured to read a symlink pointing to logs generated from my application which rotates the file using log4j2 rollingfile appender. Everything works fine till the rotation happens. after the file get rotated syslog-ng still seems to hold on to the older inode (which is not moved) and doesn't change to follow the new logs. this however does not happen in RHEL where syslog-ng recognizes the file is now rotated and moves to the new file. In both cases the sym link is always configured to point to the latest file. version details and logs from both OSs below. What am i missing here? UBUNTU - syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04) Compile-Date: Oct 24 2015 03:49:19 Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser,syslogformat,afamqp,redis,afsql,affile,afsmtp,linux-kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs,json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on symlink is pointing to the file that gets the logs. prior to rotation the process watches correctly for the file (same inodes held by my app and syslog-ng) lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log -> /x/logs/vik-test_access.log COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 1101 1542626 vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 vik-test_access.log Post rotation, syslog-ng holds on to the older file (now rotated). COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 876 1542631 e/elasticsearch-6.2.3/logs/vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log The same setup works perfectly fine in RHEL (version details below) where syslog-ng follows the new file correctly. RHEL syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722 Compile-Date: Oct 18 2012 15:17:09 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs,basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on

3 2

Random crash in the main thread
by som shukla 28 Jun '18

28 Jun '18

Hi Guys, Recently we came up with a crash in syslog-ng with following back trace. rogram terminated with signal SIGBUS, Bus error. #0 0x00007f03e00b4f7c in ?? () from /usr/lib64/libsyslog-ng.so.0 (gdb) (gdb) (gdb) (gdb) (gdb) #0 0x00007f03e00b4f7c in ?? () from /usr/lib64/libsyslog-ng.so.0 No symbol table info available. #1 0x00007f03e00b5a27 in persist_state_start () from /usr/lib64/libsyslog-ng.so.0 No symbol table info available. #2 0x00007f03e00a3d7a in cfg_initial_init () from /usr/lib64/libsyslog-ng.so.0 No symbol table info available. #3 0x00000000004028e0 in initial_init () No symbol table info available. #4 0x0000000000402a36 in main () I thought it to be similar to the this issue https://github.com/balabit/syslog-ng/issues/716 ,but I'm not sure if this is really the same. Has somebody come up with similar issue Thanks Somji

2 3