[syslog-ng] syslog-ng not following symlinks correctly on UBUNTU, works fine in RHEL

Jim Hendrick james.r.hendrick at gmail.com
Mon Jul 2 13:24:05 UTC 2018 

I'd need to look at the source. Not sure what poll does with file handles
or symlinks. Naively follow-freq() seems like it would deal with changing
inodes better but I have nothing to support that.
Jim

On Mon, Jul 2, 2018, 5:51 AM Donatello D <bluray.vik at gmail.com> wrote:

> one more observation. If i remove the follow_freq(1) on the syslog-ng
> config for this source, it seems to track file rotation clearly.
> What is the difference between specifying a follow_freq vs leaving it
> without. I read something about polling vs looking for changes every 1
> sec, but what does that exactly mean and are there any disadvantages
> of leaving out that config?
>
> On Mon, Jul 2, 2018 at 11:06 AM, Donatello D <bluray.vik at gmail.com> wrote:
> > Great point.. just checked the filesystems are different.. RHEL had
> > ext3 wherease Ubuntu is running ext4...mount options are defaults.
> > let me try to see if i can isolate this to something with ext4
> >
> > On Sun, Jul 1, 2018 at 7:24 PM, Jim Hendrick <james.r.hendrick at gmail.com>
> wrote:
> >> No clue. Maybe look at are the files on the same or different
> partitions?
> >> Are the filesystems the same type? Mount options?
> >>
> >> Weird...
> >>
> >>
> >>
> >> On Sat, Jun 30, 2018, 11:21 PM Donatello D <bluray.vik at gmail.com>
> wrote:
> >>>
> >>> @Jim - this is what i use as a  workaround already, but it is
> >>> sub-optimal, as there will be other files that are rolled over in
> >>> different intervals, so i end up reloading config multiple times.
> >>>
> >>> the real question is why does it work in RHEL and fail in UBUNTU?
> >>>
> >>>
> >>> > Date: Sat, 30 Jun 2018 20:31:59 -0400
> >>> > From: Jim Hendrick <james.r.hendrick at gmail.com>
> >>> > To: "Syslog-ng users' and developers' mailing list"
> >>> >         <syslog-ng at lists.balabit.hu>
> >>> > Subject: Re: [syslog-ng] syslog-ng not following symlinks correctly
> on
> >>> >         UBUNTU, works fine in RHEL
> >>> > Message-ID:
> >>> >
> >>> > <CANEn2idABV25G1vFa4B=WhOyuHjd3HwLMKFBHgqydH6zvH0H9w at mail.gmail.com>
> >>> > Content-Type: text/plain; charset="utf-8"
> >>> >
> >>> > hardlinks are simply additional pointers to the same inode.
> >>> > symlinks contain the name of the referenced object
> >>> >
> >>> > If you want a hack - maybe have syslog-ng reload after the file
> changes?
> >>> > (SIGHUP)
> >>> >
> >>> >
> >>> >
> >>> > On Sat, Jun 30, 2018 at 12:56 PM, Ankit Agarwal
> >>> > <ankit at travelmyheart.org>
> >>> > wrote:
> >>> >
> >>> > > Hi,
> >>> > >
> >>> > > I ran into a similar problem on Ubuntu as well.
> >>> > >
> >>> > > In my case, I was tracking the Tomcat localhost log file in
> syslog-ng
> >>> > > but
> >>> > > Tomcat creates a new log file everyday by default, and the filename
> >>> > > changes
> >>> > > (since it includes the date).
> >>> > >
> >>> > > Therefore, I periodically created a softlink to the localhost log
> file
> >>> > > where the link had a constant name. The constant name is needed
> >>> > > because I
> >>> > > obviously cannot keep changing the syslog-ng configuration to match
> >>> > > the
> >>> > > day's localhost log file name.
> >>> > >
> >>> > > I found that the softlink did not work.
> >>> > >
> >>> > > Instead I had to create a hardlink.
> >>> > >
> >>> > > This is because the softlink's modified date does not change when
> the
> >>> > > underlying file changes. The hardlink's modified date does change
> >>> > > since it
> >>> > > is pointing to the actual data. We need the modified date to change
> >>> > > for the
> >>> > > syslog-ng client to pick up new log entries.
> >>> > >
> >>> > > In my case, I periodically ran the following command via CRON in
> the
> >>> > > Tomcat logs directory:
> >>> > >
> >>> > > sudo ln -f $(ls -t localhost.* | head -1) tomcat_localhost.log
> >>> > >
> >>> > > This is to get the latest localhost log file and create the
> hardlink
> >>> > > for
> >>> > > it (overwriting the older hardlink that may have been pointing to
> the
> >>> > > previous day's localhost log file).
> >>> > >
> >>> > > I ran this every hour just to be safe.
> >>> > >
> >>> > >
> >>> > > So in your case, I think you would just need to recreate the
> hardlink
> >>> > > as
> >>> > > soon as your log file is rotated.
> >>> > >
> >>> > >
> >>> > > Hope this helps.
> >>> > >
> >>> > > Ankit
> >>> > >
> >>> > >
> >>> > >
> >>> > > ---- On Sat, 30 Jun 2018 01:13:44 -0700 *Donatello D
> >>> > > <bluray.vik at gmail.com <bluray.vik at gmail.com>>* wrote ----
> >>> > >
> >>> > > syslog-ng is configured to read a symlink pointing to logs
> generated
> >>> > > from
> >>> > > my application which rotates the file using log4j2 rollingfile
> >>> > > appender.
> >>> > > Everything works fine till the rotation happens. after the file get
> >>> > > rotated
> >>> > > syslog-ng still seems to hold on to the older inode (which is not
> >>> > > moved)
> >>> > > and doesn't change to follow the new logs. this however does not
> >>> > > happen in
> >>> > > RHEL where syslog-ng recognizes the file is now rotated and moves
> to
> >>> > > the
> >>> > > new file. In both cases the sym link is always configured to point
> to
> >>> > > the
> >>> > > latest file. version details and logs from both OSs below.
> >>> > >
> >>> > > What am i missing here?
> >>> > >
> >>> > > UBUNTU -
> >>> > > syslog-ng 3.5.6
> >>> > > Installer-Version: 3.5.6
> >>> > > Revision: 3.5.6-2.1 [@416d315] (Ubuntu/16.04)
> >>> > > Compile-Date: Oct 24 2015 03:49:19
> >>> > > Available-Modules: afsocket,afuser,tfgeoip,confgen,csvparser,
> >>> > > syslogformat,afamqp,redis,afsql,affile,afsmtp,linux-
> >>> > > kmsg-format,dbparser,system-source,cryptofuncs,basicfuncs,
> >>> > > json-plugin,afprog,afsocket-tls,afstomp,afsocket-notls,afmongodb
> >>> > > Enable-Debug: off
> >>> > > Enable-GProf: off
> >>> > > Enable-Memtrace: off
> >>> > > Enable-IPv6: on
> >>> > > Enable-Spoof-Source: on
> >>> > > Enable-TCP-Wrapper: on
> >>> > > Enable-Linux-Caps: on
> >>> > > Enable-Pcre: on
> >>> > >
> >>> > > symlink is pointing to the file that gets the logs. prior to
> rotation
> >>> > > the
> >>> > > process watches correctly for the file (same inodes held by my app
> and
> >>> > > syslog-ng)
> >>> > >
> >>> > > lrwxrwxrwx 1 root root 56 Jun 29 08:44 node1-access.log ->
> >>> > > /x/logs/vik-test_access.log
> >>> > >
> >>> > > COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
> >>> > > java      11032       vikram 53w   REG    8,1     1101 1542626
> >>> > > vik-test_access.log
> >>> > > syslog-ng 21661       root    9r   REG    8,1     1101 1542626
> >>> > > vik-test_access.log
> >>> > >
> >>> > >
> >>> > > Post rotation, syslog-ng holds on to the older file (now rotated).
> >>> > >
> >>> > > COMMAND     PID       USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
> >>> > > java      11032       vikram  53w   REG    8,1      876 1542631
> >>> > > e/elasticsearch-6.2.3/logs/vik-test_access.log
> >>> > > syslog-ng 21661       root    9r   REG    8,1     1101 1542626
> >>> > > e/elasticsearch-6.2.3/logs/vik-test_access-2018-06-30.log
> >>> > >
> >>> > > The same setup works perfectly fine in RHEL (version details below)
> >>> > > where
> >>> > > syslog-ng follows the new file correctly.
> >>> > >
> >>> > > RHEL
> >>> > > syslog-ng 3.3.5
> >>> > > Installer-Version: 3.3.5
> >>> > > Revision: ssh+git://bazsi@git.balabit
> //var/scm/git/syslog-ng/syslog-
> >>> > >
> ng-ose--mainline--3.3--master#d5d607c05251b38e821efe27bc46ac8db78dd722
> >>> > > Compile-Date: Oct 18 2012 15:17:09
> >>> > > Default-Modules: affile,afprog,afsocket,afuser,
> >>> > > basicfuncs,csvparser,dbparser,syslogformat
> >>> > > Available-Modules:
> afprog,afsocket-tls,dbparser,confgen,convertfuncs,
> >>> > >
> >>> > >
> basicfuncs,afsocket,afmongodb,csvparser,affile,dummy,syslogformat,afuser
> >>> > > Enable-Debug: off
> >>> > > Enable-GProf: off
> >>> > > Enable-Memtrace: off
> >>> > > Enable-IPv6: on
> >>> > > Enable-Spoof-Source: off
> >>> > > Enable-TCP-Wrapper: on
> >>> > > Enable-Linux-Caps: off
> >>> > > Enable-Pcre: on
> >>> > >
> >>> > > ____________________________________________________________
> >>> > > __________________
> >>> > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>> > > Documentation: http://www.balabit.com/support/documentation/?
> >>> > > product=syslog-ng
> >>> > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>> > >
> >>> > >
> >>> > >
> >>> > >
> >>> > > ____________________________________________________________
> >>> > > __________________
> >>> > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>> > > Documentation: http://www.balabit.com/support/documentation/?
> >>> > > product=syslog-ng
> >>> > > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>> > >
> >>> > >
> >>> > >
> >>> > -------------- next part --------------
> >>> > An HTML attachment was scrubbed...
> >>> > URL:
> >>> > <
> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180630/db683a38/attachment.html
> >
> >>> >
> >>> > ------------------------------
> >>> >
> >>> > Subject: Digest Footer
> >>> >
> >>> > _______________________________________________
> >>> > syslog-ng maillist  -  syslog-ng at lists.balabit.hu
> >>> > https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>> >
> >>> >
> >>> > ------------------------------
> >>> >
> >>> > End of syslog-ng Digest, Vol 159, Issue 1
> >>> > *****************************************
> >>>
> >>>
> ______________________________________________________________________________
> >>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >>> Documentation:
> >>> http://www.balabit.com/support/documentation/?product=syslog-ng
> >>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>>
> >>
> >>
> ______________________________________________________________________________
> >> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> >> Documentation:
> >> http://www.balabit.com/support/documentation/?product=syslog-ng
> >> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >>
> >>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20180702/5ffe5290/attachment-0001.html>

More information about the syslog-ng mailing list