[syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL

Sun Jul 1 03:26:51 UTC 2018

I suspect it is more related to the filesystem and options.

1. Are you using a different fileystem (xfs vs ext3 or NFS)
2. Are there different mount options for the filesystem?

Evan.

On 06/30/2018 08:21 PM, Donatello D wrote:
> @Jim - this is what i use as a  workaround already, but it is
> sub-optimal, as there will be other files that are rolled over in
> different intervals, so i end up reloading config multiple times.
>
> the real question is why does it work in RHEL and fail in UBUNTU?
>
>
>> Date: Sat, 30 Jun 2018 20:31:59 -0400
>> From: Jim Hendrick <james.r.hendrick at gmail.com>
>> To: "Syslog-ng users' and developers' mailing list"
>>          <syslog-ng at lists.balabit.hu>
>> Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly on
>>          UBUNTU, works fine in RHEL
>> Message-ID:
>>          <CANEn2idABV25G1vFa4B=WhOyuHjd3HwLMKFBHgqydH6zvH0H9w at mail.gmail.com>
>> Content-Type: text/plain; charset="utf-8"
>>
>> hardlinks are simply additional pointers to the same inode.
>> symlinks contain the name of the referenced object
>>
>> If you want a hack - maybe have syslog-ng reload after the file changes?
>> (SIGHUP)
>>
>>
>>
>> On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal <ankit at travelmyheart.org>
>> wrote:
>>
>>> Hi,
>>>
>>> I ran into a similar problem on Ubuntu as well.
>>>
>>> In my case, I was tracking the Tomcat localhost log file in syslog-ng but
>>> Tomcat creates a new log file everyday by default, and the filename changes
>>> (since it includes the date).
>>>
>>> Therefore, I periodically created a softlink to the localhost log file
>>> where the link had a constant name. The constant name is needed because I
>>> obviously cannot keep changing the syslog-ng configuration to match the
>>> day's localhost log file name.
>>>
>>> I found that the softlink did not work.
>>>
>>> Instead I had to create a hardlink.
>>>
>>> This is because the softlink's modified date does not change when the
>>> underlying file changes. The hardlink's modified date does change since it
>>> is pointing to the actual data. We need the modified date to change for the
>>> syslog-ng client to pick up new log entries.
>>>
>>> In my case, I periodically ran the following command via CRON in the
>>> Tomcat logs directory:
>>>
>>> sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log
>>>
>>> This is to get the latest localhost log file and create the hardlink for
>>> it (overwriting the older hardlink that may have been pointing to the
>>> previous day's localhost log file).
>>>
>>> I ran this every hour just to be safe.
>>>
>>>
>>> So in your case, I think you would just need to recreate the hardlink as
>>> soon as your log file is rotated.
>>>
>>>
>>> Hope this helps.
>>>
>>> Ankit
>>>
>>>
>>>
>>> ---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D
>>> <bluray.vik at gmail.com <bluray.vik at gmail.com>>* wrote ----
>>>
>>> syslog-ng is configured to read a symlink pointing to logs generated from
>>> my application which rotates the file using log4j2 rollingfile appender.
>>> Everything works fine till the rotation happens. after the file get rotated
>>> syslog-ng still seems to hold on to the older inode (which is not moved)
>>> and doesn't change to follow the new logs. this however does not happen in
>>> RHEL where syslog-ng recognizes the file is now rotated and moves to the
>>> new file. In both cases the sym link is always configured to point to the
>>> latest file. version details and logs from both OSs below.
>>>
>>> What am i missing here?
>>>
>>> UBUNTU -
>>> syslog-ng 3.5.6
>>> Installer-Version: 3.5.6
>>> Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04)
>>> Compile-Date: Oct 24 2015 03:49:19
>>> Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser,
>>> syslogformat,afamqp,redis,afsql,affile,afsmtp,linux-
>>> kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs,
>>> json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb
>>> Enable-Debug: off
>>> Enable-GProf: off
>>> Enable-Memtrace: off
>>> Enable-IPv6: on
>>> Enable-Spoof-Source: on
>>> Enable-TCP-Wrapper: on
>>> Enable-Linux-Caps: on
>>> Enable-Pcre: on
>>>
>>> symlink is pointing to the file that gets the logs. prior to rotation the
>>> process watches correctly for the file (same inodes held by my app and
>>> syslog-ng)
>>>
>>> lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log ->
>>> /x/logs/vik-test_access.log
>>>
>>> COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
>>> java      11032       vikram 53w   REG    8,1     1101 1542626
>>> vik-test_access.log
>>> syslog-ng 21661       root    9r   REG    8,1     1101 1542626
>>> vik-test_access.log
>>>
>>>
>>> Post rotation, syslog-ng holds on to the older file (now rotated).
>>>
>>> COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
>>> java      11032       vikram  53w   REG    8,1      876 1542631
>>> e/elasticsearch-6.2.3/logs/vik-test_access.log
>>> syslog-ng 21661       root    9r   REG    8,1     1101 1542626
>>> e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log
>>>
>>> The same setup works perfectly fine in RHEL (version details below) where
>>> syslog-ng follows the new file correctly.
>>>
>>> RHEL
>>> syslog-ng 3.3.5
>>> Installer-Version: 3.3.5
>>> Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-
>>> ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722
>>> Compile-Date: Oct 18 2012 15:17:09
>>> Default-Modules: affile,afprog,afsocket,afuser,
>>> basicfuncs,csvparser,dbparser,syslogformat
>>> Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs,
>>> basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser
>>> Enable-Debug: off
>>> Enable-GProf: off
>>> Enable-Memtrace: off
>>> Enable-IPv6: on
>>> Enable-Spoof-Source: off
>>> Enable-TCP-Wrapper: on
>>> Enable-Linux-Caps: off
>>> Enable-Pcre: on
>>>
>>>