Re: [syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL
@Jim - this is what i use as a workaround already, but it is sub-optimal, as there will be other files that are rolled over in different intervals, so i end up reloading config multiple times. the real question is why does it work in RHEL and fail in UBUNTU?
Date: Sat, 30 Jun 2018 20:31:59 -0400 From: Jim Hendrick <james.r.hendrick@gmail.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL Message-ID: <CANEn2idABV25G1vFa4B=WhOyuHjd3HwLMKFBHgqydH6zvH0H9w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
hardlinks are simply additional pointers to the same inode. symlinks contain the name of the referenced object
If you want a hack - maybe have syslog-ng reload after the file changes? (SIGHUP)
On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal <ankit@travelmyheart.org> wrote:
Hi,
I ran into a similar problem on Ubuntu as well.
In my case, I was tracking the Tomcat localhost log file in syslog-ng but Tomcat creates a new log file everyday by default, and the filename changes (since it includes the date).
Therefore, I periodically created a softlink to the localhost log file where the link had a constant name. The constant name is needed because I obviously cannot keep changing the syslog-ng configuration to match the day's localhost log file name.
I found that the softlink did not work.
Instead I had to create a hardlink.
This is because the softlink's modified date does not change when the underlying file changes. The hardlink's modified date does change since it is pointing to the actual data. We need the modified date to change for the syslog-ng client to pick up new log entries.
In my case, I periodically ran the following command via CRON in the Tomcat logs directory:
sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log
This is to get the latest localhost log file and create the hardlink for it (overwriting the older hardlink that may have been pointing to the previous day's localhost log file).
I ran this every hour just to be safe.
So in your case, I think you would just need to recreate the hardlink as soon as your log file is rotated.
Hope this helps.
Ankit
---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D <bluray.vik@gmail.com <bluray.vik@gmail.com>>* wrote ----
syslog-ng is configured to read a symlink pointing to logs generated from my application which rotates the file using log4j2 rollingfile appender. Everything works fine till the rotation happens. after the file get rotated syslog-ng still seems to hold on to the older inode (which is not moved) and doesn't change to follow the new logs. this however does not happen in RHEL where syslog-ng recognizes the file is now rotated and moves to the new file. In both cases the sym link is always configured to point to the latest file. version details and logs from both OSs below.
What am i missing here?
UBUNTU - syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04) Compile-Date: Oct 24 2015 03:49:19 Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser, syslogformat,afamqp,redis,afsql,affile,afsmtp,linux- kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs, json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on
symlink is pointing to the file that gets the logs. prior to rotation the process watches correctly for the file (same inodes held by my app and syslog-ng)
lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log -> /x/logs/vik-test_access.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 1101 1542626 vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 vik-test_access.log
Post rotation, syslog-ng holds on to the older file (now rotated).
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 876 1542631 e/elasticsearch-6.2.3/logs/vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log
The same setup works perfectly fine in RHEL (version details below) where syslog-ng follows the new file correctly.
RHEL syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog- ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722 Compile-Date: Oct 18 2012 15:17:09 Default-Modules: affile,afprog,afsocket,afuser, basicfuncs,csvparser,dbparser,syslogformat Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs, basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I suspect it is more related to the filesystem and options. 1. Are you using a different fileystem (xfs vs ext3 or NFS) 2. Are there different mount options for the filesystem? Evan. On 06/30/2018 08:21 PM, Donatello D wrote:
@Jim - this is what i use as a workaround already, but it is sub-optimal, as there will be other files that are rolled over in different intervals, so i end up reloading config multiple times.
the real question is why does it work in RHEL and fail in UBUNTU?
Date: Sat, 30 Jun 2018 20:31:59 -0400 From: Jim Hendrick <james.r.hendrick@gmail.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL Message-ID: <CANEn2idABV25G1vFa4B=WhOyuHjd3HwLMKFBHgqydH6zvH0H9w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
hardlinks are simply additional pointers to the same inode. symlinks contain the name of the referenced object
If you want a hack - maybe have syslog-ng reload after the file changes? (SIGHUP)
On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal <ankit@travelmyheart.org> wrote:
Hi,
I ran into a similar problem on Ubuntu as well.
In my case, I was tracking the Tomcat localhost log file in syslog-ng but Tomcat creates a new log file everyday by default, and the filename changes (since it includes the date).
Therefore, I periodically created a softlink to the localhost log file where the link had a constant name. The constant name is needed because I obviously cannot keep changing the syslog-ng configuration to match the day's localhost log file name.
I found that the softlink did not work.
Instead I had to create a hardlink.
This is because the softlink's modified date does not change when the underlying file changes. The hardlink's modified date does change since it is pointing to the actual data. We need the modified date to change for the syslog-ng client to pick up new log entries.
In my case, I periodically ran the following command via CRON in the Tomcat logs directory:
sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log
This is to get the latest localhost log file and create the hardlink for it (overwriting the older hardlink that may have been pointing to the previous day's localhost log file).
I ran this every hour just to be safe.
So in your case, I think you would just need to recreate the hardlink as soon as your log file is rotated.
Hope this helps.
Ankit
---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D <bluray.vik@gmail.com <bluray.vik@gmail.com>>* wrote ----
syslog-ng is configured to read a symlink pointing to logs generated from my application which rotates the file using log4j2 rollingfile appender. Everything works fine till the rotation happens. after the file get rotated syslog-ng still seems to hold on to the older inode (which is not moved) and doesn't change to follow the new logs. this however does not happen in RHEL where syslog-ng recognizes the file is now rotated and moves to the new file. In both cases the sym link is always configured to point to the latest file. version details and logs from both OSs below.
What am i missing here?
UBUNTU - syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04) Compile-Date: Oct 24 2015 03:49:19 Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser, syslogformat,afamqp,redis,afsql,affile,afsmtp,linux- kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs, json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on
symlink is pointing to the file that gets the logs. prior to rotation the process watches correctly for the file (same inodes held by my app and syslog-ng)
lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log -> /x/logs/vik-test_access.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 1101 1542626 vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 vik-test_access.log
Post rotation, syslog-ng holds on to the older file (now rotated).
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 876 1542631 e/elasticsearch-6.2.3/logs/vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log
The same setup works perfectly fine in RHEL (version details below) where syslog-ng follows the new file correctly.
RHEL syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog- ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722 Compile-Date: Oct 18 2012 15:17:09 Default-Modules: affile,afprog,afsocket,afuser, basicfuncs,csvparser,dbparser,syslogformat Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs, basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on
the real question is why does it work in RHEL and fail in UBUNTU?
The difference might come from that you are using different versions of syslog-ng on the two systems:
UBUNTU - syslog-ng 3.5.6 RHEL syslog-ng 3.3.5
The constant name is needed because I obviously cannot keep changing the syslog-ng configuration to match the day's localhost log file name.
You can avoid such hack. Please check the recent addition to syslog-ng: wildcard filesource. You need to upgrade to 3.16 though. https://syslog-ng.com/documents/html/syslog-ng-ose-latest-guides/en/syslog-n... On Sun, Jul 1, 2018 at 5:26 AM Evan Rempel <erempel@uvic.ca> wrote:
I suspect it is more related to the filesystem and options.
1. Are you using a different fileystem (xfs vs ext3 or NFS) 2. Are there different mount options for the filesystem?
Evan.
On 06/30/2018 08:21 PM, Donatello D wrote:
@Jim - this is what i use as a workaround already, but it is sub-optimal, as there will be other files that are rolled over in different intervals, so i end up reloading config multiple times.
the real question is why does it work in RHEL and fail in UBUNTU?
Date: Sat, 30 Jun 2018 20:31:59 -0400 From: Jim Hendrick <james.r.hendrick@gmail.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL Message-ID: <CANEn2idABV25G1vFa4B= WhOyuHjd3HwLMKFBHgqydH6zvH0H9w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
hardlinks are simply additional pointers to the same inode. symlinks contain the name of the referenced object
If you want a hack - maybe have syslog-ng reload after the file changes? (SIGHUP)
On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal < ankit@travelmyheart.org> wrote:
Hi,
I ran into a similar problem on Ubuntu as well.
In my case, I was tracking the Tomcat localhost log file in syslog-ng but Tomcat creates a new log file everyday by default, and the filename changes (since it includes the date).
Therefore, I periodically created a softlink to the localhost log file where the link had a constant name. The constant name is needed because I obviously cannot keep changing the syslog-ng configuration to match the day's localhost log file name.
I found that the softlink did not work.
Instead I had to create a hardlink.
This is because the softlink's modified date does not change when the underlying file changes. The hardlink's modified date does change since it is pointing to the actual data. We need the modified date to change for the syslog-ng client to pick up new log entries.
In my case, I periodically ran the following command via CRON in the Tomcat logs directory:
sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log
This is to get the latest localhost log file and create the hardlink for it (overwriting the older hardlink that may have been pointing to the previous day's localhost log file).
I ran this every hour just to be safe.
So in your case, I think you would just need to recreate the hardlink as soon as your log file is rotated.
Hope this helps.
Ankit
---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D <bluray.vik@gmail.com <bluray.vik@gmail.com>>* wrote ----
syslog-ng is configured to read a symlink pointing to logs generated from my application which rotates the file using log4j2 rollingfile appender. Everything works fine till the rotation happens. after the file get rotated syslog-ng still seems to hold on to the older inode (which is not moved) and doesn't change to follow the new logs. this however does not happen in RHEL where syslog-ng recognizes the file is now rotated and moves to the new file. In both cases the sym link is always configured to point to the latest file. version details and logs from both OSs below.
What am i missing here?
UBUNTU - syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04) Compile-Date: Oct 24 2015 03:49:19 Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser, syslogformat,afamqp,redis,afsql,affile,afsmtp,linux- kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs, json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on
symlink is pointing to the file that gets the logs. prior to rotation the process watches correctly for the file (same inodes held by my app and syslog-ng)
lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log -> /x/logs/vik-test_access.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 1101 1542626 vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 vik-test_access.log
Post rotation, syslog-ng holds on to the older file (now rotated).
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 876 1542631 e/elasticsearch-6.2.3/logs/vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log
The same setup works perfectly fine in RHEL (version details below) where syslog-ng follows the new file correctly.
RHEL syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog- ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722 Compile-Date: Oct 18 2012 15:17:09 Default-Modules: affile,afprog,afsocket,afuser, basicfuncs,csvparser,dbparser,syslogformat Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs,
basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser
Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
No clue. Maybe look at are the files on the same or different partitions? Are the filesystems the same type? Mount options? Weird... On Sat, Jun 30, 2018, 11:21 PM Donatello D <bluray.vik@gmail.com> wrote:
@Jim - this is what i use as a workaround already, but it is sub-optimal, as there will be other files that are rolled over in different intervals, so i end up reloading config multiple times.
the real question is why does it work in RHEL and fail in UBUNTU?
Date: Sat, 30 Jun 2018 20:31:59 -0400 From: Jim Hendrick <james.r.hendrick@gmail.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL Message-ID: <CANEn2idABV25G1vFa4B= WhOyuHjd3HwLMKFBHgqydH6zvH0H9w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
hardlinks are simply additional pointers to the same inode. symlinks contain the name of the referenced object
If you want a hack - maybe have syslog-ng reload after the file changes? (SIGHUP)
On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal <ankit@travelmyheart.org
wrote:
Hi,
I ran into a similar problem on Ubuntu as well.
In my case, I was tracking the Tomcat localhost log file in syslog-ng but Tomcat creates a new log file everyday by default, and the filename changes (since it includes the date).
Therefore, I periodically created a softlink to the localhost log file where the link had a constant name. The constant name is needed because I obviously cannot keep changing the syslog-ng configuration to match the day's localhost log file name.
I found that the softlink did not work.
Instead I had to create a hardlink.
This is because the softlink's modified date does not change when the underlying file changes. The hardlink's modified date does change since it is pointing to the actual data. We need the modified date to change for the syslog-ng client to pick up new log entries.
In my case, I periodically ran the following command via CRON in the Tomcat logs directory:
sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log
This is to get the latest localhost log file and create the hardlink for it (overwriting the older hardlink that may have been pointing to the previous day's localhost log file).
I ran this every hour just to be safe.
So in your case, I think you would just need to recreate the hardlink as soon as your log file is rotated.
Hope this helps.
Ankit
---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D <bluray.vik@gmail.com <bluray.vik@gmail.com>>* wrote ----
syslog-ng is configured to read a symlink pointing to logs generated from my application which rotates the file using log4j2 rollingfile appender. Everything works fine till the rotation happens. after the file get rotated syslog-ng still seems to hold on to the older inode (which is not moved) and doesn't change to follow the new logs. this however does not happen in RHEL where syslog-ng recognizes the file is now rotated and moves to the new file. In both cases the sym link is always configured to point to the latest file. version details and logs from both OSs below.
What am i missing here?
UBUNTU - syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04) Compile-Date: Oct 24 2015 03:49:19 Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser, syslogformat,afamqp,redis,afsql,affile,afsmtp,linux- kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs, json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on
symlink is pointing to the file that gets the logs. prior to rotation the process watches correctly for the file (same inodes held by my app and syslog-ng)
lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log -> /x/logs/vik-test_access.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 1101 1542626 vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 vik-test_access.log
Post rotation, syslog-ng holds on to the older file (now rotated).
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 876 1542631 e/elasticsearch-6.2.3/logs/vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log
The same setup works perfectly fine in RHEL (version details below) where syslog-ng follows the new file correctly.
RHEL syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog- ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722 Compile-Date: Oct 18 2012 15:17:09 Default-Modules: affile,afprog,afsocket,afuser, basicfuncs,csvparser,dbparser,syslogformat Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs,
basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser
Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Great point.. just checked the filesystems are different.. RHEL had ext3 wherease Ubuntu is running ext4...mount options are defaults. let me try to see if i can isolate this to something with ext4 On Sun, Jul 1, 2018 at 7:24 PM, Jim Hendrick <james.r.hendrick@gmail.com> wrote:
No clue. Maybe look at are the files on the same or different partitions? Are the filesystems the same type? Mount options?
Weird...
On Sat, Jun 30, 2018, 11:21 PM Donatello D <bluray.vik@gmail.com> wrote:
@Jim - this is what i use as a workaround already, but it is sub-optimal, as there will be other files that are rolled over in different intervals, so i end up reloading config multiple times.
the real question is why does it work in RHEL and fail in UBUNTU?
Date: Sat, 30 Jun 2018 20:31:59 -0400 From: Jim Hendrick <james.r.hendrick@gmail.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL Message-ID:
<CANEn2idABV25G1vFa4B=WhOyuHjd3HwLMKFBHgqydH6zvH0H9w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
hardlinks are simply additional pointers to the same inode. symlinks contain the name of the referenced object
If you want a hack - maybe have syslog-ng reload after the file changes? (SIGHUP)
On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal <ankit@travelmyheart.org> wrote:
Hi,
I ran into a similar problem on Ubuntu as well.
In my case, I was tracking the Tomcat localhost log file in syslog-ng but Tomcat creates a new log file everyday by default, and the filename changes (since it includes the date).
Therefore, I periodically created a softlink to the localhost log file where the link had a constant name. The constant name is needed because I obviously cannot keep changing the syslog-ng configuration to match the day's localhost log file name.
I found that the softlink did not work.
Instead I had to create a hardlink.
This is because the softlink's modified date does not change when the underlying file changes. The hardlink's modified date does change since it is pointing to the actual data. We need the modified date to change for the syslog-ng client to pick up new log entries.
In my case, I periodically ran the following command via CRON in the Tomcat logs directory:
sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log
This is to get the latest localhost log file and create the hardlink for it (overwriting the older hardlink that may have been pointing to the previous day's localhost log file).
I ran this every hour just to be safe.
So in your case, I think you would just need to recreate the hardlink as soon as your log file is rotated.
Hope this helps.
Ankit
---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D <bluray.vik@gmail.com <bluray.vik@gmail.com>>* wrote ----
syslog-ng is configured to read a symlink pointing to logs generated from my application which rotates the file using log4j2 rollingfile appender. Everything works fine till the rotation happens. after the file get rotated syslog-ng still seems to hold on to the older inode (which is not moved) and doesn't change to follow the new logs. this however does not happen in RHEL where syslog-ng recognizes the file is now rotated and moves to the new file. In both cases the sym link is always configured to point to the latest file. version details and logs from both OSs below.
What am i missing here?
UBUNTU - syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04) Compile-Date: Oct 24 2015 03:49:19 Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser, syslogformat,afamqp,redis,afsql,affile,afsmtp,linux- kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs, json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on
symlink is pointing to the file that gets the logs. prior to rotation the process watches correctly for the file (same inodes held by my app and syslog-ng)
lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log -> /x/logs/vik-test_access.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 1101 1542626 vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 vik-test_access.log
Post rotation, syslog-ng holds on to the older file (now rotated).
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 876 1542631 e/elasticsearch-6.2.3/logs/vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log
The same setup works perfectly fine in RHEL (version details below) where syslog-ng follows the new file correctly.
RHEL syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog- ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722 Compile-Date: Oct 18 2012 15:17:09 Default-Modules: affile,afprog,afsocket,afuser, basicfuncs,csvparser,dbparser,syslogformat Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs,
basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
one more observation. If i remove the follow_freq(1) on the syslog-ng config for this source, it seems to track file rotation clearly. What is the difference between specifying a follow_freq vs leaving it without. I read something about polling vs looking for changes every 1 sec, but what does that exactly mean and are there any disadvantages of leaving out that config? On Mon, Jul 2, 2018 at 11:06 AM, Donatello D <bluray.vik@gmail.com> wrote:
Great point.. just checked the filesystems are different.. RHEL had ext3 wherease Ubuntu is running ext4...mount options are defaults. let me try to see if i can isolate this to something with ext4
On Sun, Jul 1, 2018 at 7:24 PM, Jim Hendrick <james.r.hendrick@gmail.com> wrote:
No clue. Maybe look at are the files on the same or different partitions? Are the filesystems the same type? Mount options?
Weird...
On Sat, Jun 30, 2018, 11:21 PM Donatello D <bluray.vik@gmail.com> wrote:
@Jim - this is what i use as a workaround already, but it is sub-optimal, as there will be other files that are rolled over in different intervals, so i end up reloading config multiple times.
the real question is why does it work in RHEL and fail in UBUNTU?
Date: Sat, 30 Jun 2018 20:31:59 -0400 From: Jim Hendrick <james.r.hendrick@gmail.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL Message-ID:
<CANEn2idABV25G1vFa4B=WhOyuHjd3HwLMKFBHgqydH6zvH0H9w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
hardlinks are simply additional pointers to the same inode. symlinks contain the name of the referenced object
If you want a hack - maybe have syslog-ng reload after the file changes? (SIGHUP)
On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal <ankit@travelmyheart.org> wrote:
Hi,
I ran into a similar problem on Ubuntu as well.
In my case, I was tracking the Tomcat localhost log file in syslog-ng but Tomcat creates a new log file everyday by default, and the filename changes (since it includes the date).
Therefore, I periodically created a softlink to the localhost log file where the link had a constant name. The constant name is needed because I obviously cannot keep changing the syslog-ng configuration to match the day's localhost log file name.
I found that the softlink did not work.
Instead I had to create a hardlink.
This is because the softlink's modified date does not change when the underlying file changes. The hardlink's modified date does change since it is pointing to the actual data. We need the modified date to change for the syslog-ng client to pick up new log entries.
In my case, I periodically ran the following command via CRON in the Tomcat logs directory:
sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log
This is to get the latest localhost log file and create the hardlink for it (overwriting the older hardlink that may have been pointing to the previous day's localhost log file).
I ran this every hour just to be safe.
So in your case, I think you would just need to recreate the hardlink as soon as your log file is rotated.
Hope this helps.
Ankit
---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D <bluray.vik@gmail.com <bluray.vik@gmail.com>>* wrote ----
syslog-ng is configured to read a symlink pointing to logs generated from my application which rotates the file using log4j2 rollingfile appender. Everything works fine till the rotation happens. after the file get rotated syslog-ng still seems to hold on to the older inode (which is not moved) and doesn't change to follow the new logs. this however does not happen in RHEL where syslog-ng recognizes the file is now rotated and moves to the new file. In both cases the sym link is always configured to point to the latest file. version details and logs from both OSs below.
What am i missing here?
UBUNTU - syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04) Compile-Date: Oct 24 2015 03:49:19 Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser, syslogformat,afamqp,redis,afsql,affile,afsmtp,linux- kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs, json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on
symlink is pointing to the file that gets the logs. prior to rotation the process watches correctly for the file (same inodes held by my app and syslog-ng)
lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log -> /x/logs/vik-test_access.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 1101 1542626 vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 vik-test_access.log
Post rotation, syslog-ng holds on to the older file (now rotated).
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 876 1542631 e/elasticsearch-6.2.3/logs/vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log
The same setup works perfectly fine in RHEL (version details below) where syslog-ng follows the new file correctly.
RHEL syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog- ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722 Compile-Date: Oct 18 2012 15:17:09 Default-Modules: affile,afprog,afsocket,afuser, basicfuncs,csvparser,dbparser,syslogformat Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs,
basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
I'd need to look at the source. Not sure what poll does with file handles or symlinks. Naively follow-freq() seems like it would deal with changing inodes better but I have nothing to support that. Jim On Mon, Jul 2, 2018, 5:51 AM Donatello D <bluray.vik@gmail.com> wrote:
one more observation. If i remove the follow_freq(1) on the syslog-ng config for this source, it seems to track file rotation clearly. What is the difference between specifying a follow_freq vs leaving it without. I read something about polling vs looking for changes every 1 sec, but what does that exactly mean and are there any disadvantages of leaving out that config?
On Mon, Jul 2, 2018 at 11:06 AM, Donatello D <bluray.vik@gmail.com> wrote:
Great point.. just checked the filesystems are different.. RHEL had ext3 wherease Ubuntu is running ext4...mount options are defaults. let me try to see if i can isolate this to something with ext4
On Sun, Jul 1, 2018 at 7:24 PM, Jim Hendrick <james.r.hendrick@gmail.com> wrote:
No clue. Maybe look at are the files on the same or different partitions? Are the filesystems the same type? Mount options?
Weird...
On Sat, Jun 30, 2018, 11:21 PM Donatello D <bluray.vik@gmail.com> wrote:
@Jim - this is what i use as a workaround already, but it is sub-optimal, as there will be other files that are rolled over in different intervals, so i end up reloading config multiple times.
the real question is why does it work in RHEL and fail in UBUNTU?
Date: Sat, 30 Jun 2018 20:31:59 -0400 From: Jim Hendrick <james.r.hendrick@gmail.com> To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly
on
UBUNTU, works fine in RHEL Message-ID:
<CANEn2idABV25G1vFa4B=WhOyuHjd3HwLMKFBHgqydH6zvH0H9w@mail.gmail.com> Content-Type: text/plain; charset="utf-8"
hardlinks are simply additional pointers to the same inode. symlinks contain the name of the referenced object
If you want a hack - maybe have syslog-ng reload after the file
changes?
(SIGHUP)
On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal <ankit@travelmyheart.org> wrote:
Hi,
I ran into a similar problem on Ubuntu as well.
In my case, I was tracking the Tomcat localhost log file in syslog-ng but Tomcat creates a new log file everyday by default, and the filename changes (since it includes the date).
Therefore, I periodically created a softlink to the localhost log file where the link had a constant name. The constant name is needed because I obviously cannot keep changing the syslog-ng configuration to match the day's localhost log file name.
I found that the softlink did not work.
Instead I had to create a hardlink.
This is because the softlink's modified date does not change when the underlying file changes. The hardlink's modified date does change since it is pointing to the actual data. We need the modified date to change for the syslog-ng client to pick up new log entries.
In my case, I periodically ran the following command via CRON in the Tomcat logs directory:
sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log
This is to get the latest localhost log file and create the hardlink for it (overwriting the older hardlink that may have been pointing to the previous day's localhost log file).
I ran this every hour just to be safe.
So in your case, I think you would just need to recreate the hardlink as soon as your log file is rotated.
Hope this helps.
Ankit
---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D <bluray.vik@gmail.com <bluray.vik@gmail.com>>* wrote ----
syslog-ng is configured to read a symlink pointing to logs generated from my application which rotates the file using log4j2 rollingfile appender. Everything works fine till the rotation happens. after the file get rotated syslog-ng still seems to hold on to the older inode (which is not moved) and doesn't change to follow the new logs. this however does not happen in RHEL where syslog-ng recognizes the file is now rotated and moves to the new file. In both cases the sym link is always configured to point to the latest file. version details and logs from both OSs below.
What am i missing here?
UBUNTU - syslog-ng 3.5.6 Installer-Version: 3.5.6 Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04) Compile-Date: Oct 24 2015 03:49:19 Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser, syslogformat,afamqp,redis,afsql,affile,afsmtp,linux- kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs, json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on
symlink is pointing to the file that gets the logs. prior to rotation the process watches correctly for the file (same inodes held by my app and syslog-ng)
lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log -> /x/logs/vik-test_access.log
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 1101 1542626 vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 vik-test_access.log
Post rotation, syslog-ng holds on to the older file (now rotated).
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 11032 vikram 53w REG 8,1 876 1542631 e/elasticsearch-6.2.3/logs/vik-test_access.log syslog-ng 21661 root 9r REG 8,1 1101 1542626 e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log
The same setup works perfectly fine in RHEL (version details below) where syslog-ng follows the new file correctly.
RHEL syslog-ng 3.3.5 Installer-Version: 3.3.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-
ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722
Compile-Date: Oct 18 2012 15:17:09 Default-Modules: affile,afprog,afsocket,afuser, basicfuncs,csvparser,dbparser,syslogformat Available-Modules: afprog,afsocket-tls,dbparser,confgen,convertfuncs,
basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser
Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: on
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Antal Nemes
-
Donatello D
-
Evan Rempel
-
Jim Hendrick