syslog-ng 3.3.3 repeatedly writes same message to local file when forwarding enabled
Hi, I have a problem with syslog-ng 3.3.3. When I have forwarding enabled to a remote syslog server (via UDP) syslog-ng repeatedly writes the same message(s) to the log file and only stops when the disk is full. Using tcpdump on the remote server, I don't see any data arrive from the syslog-ng server so forwarding is not working either. When I remove the forwarding part of the config file the local file is written correctly (ie once). If I remove the local file part from the config file and only enable the forwarding, I see syslog-ng take all the CPU time. I never see any syslog messages arrive at the remote syslog server. I tried: 1) disabling IPv6 - no change 2) running outside the chroot jail - no change 3) running as userid root - no change Does anyone have any idea what would cause this? Debug info below. The environment is: RedHat AS 4.8 (linux 2.6.9-89.ELsmp) on vmware ESXi 4.1.0 All required software built and installed in /usr/local/ : eventlog_0.2.12.tar.gz gettext-0.18.1.1.tar.gz glib-2.29.90.tar.bz2 libdbi-0.8.4.tar.gz libdbi-drivers-0.8.3.tar.gz libffi-3.0.9.tar.gz libnet-0.10.11.tar.gz pkg-config-0.26.tar.gz Python-2.7.2.tar.bz2 zlib-1.2.5.tar.bz2 syslog-ng_3.3.3.tar.gz syslog-ng is running chroot() in directory /data as user syslogng:sysadmins and listens on port 1514. iptables redirects any incoming port 514 traffic to 1514. The required /usr/local/ directories are mounted (-o bind) under /data. syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Dec 8 2011 17:46:40 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: convertfuncs,affile,afmongodb,dummy,basicfuncs,csvparser,confgen,afsql,syslogformat,afuser,afsocket,afprog,afsocket-notls,dbparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: off Config file: @version: 3.3 source s_udp { udp(ip("0.0.0.0") port(1514)); }; destination file1 { file("/log/network.log" owner(syslogng)group(sysops) perm(0640) flags(no-multi-line)); }; destination NeDi { udp("192.168.0.7" port(514)); }; log { source(s_udp); destination(file1); }; # enabling the line below breaks logging to the file above log { source(s_udp); destination(NeDi); }; Debug: # /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng.conf --chroot=/data --user=syslogng --group=sysadmins --persist-file=/log/syslog-ng.persist --foreground --process-mode=foreground --stderr --debug nanosleep() is not accurate enough to introduce minor stalls on the reader side, multi-threaded performance may be affected; Trying to open module; module='affile', filename='/usr/local/lib/syslog-ng/libaffile.so' Trying to open module; module='afprog', filename='/usr/local/lib/syslog-ng/libafprog.so' Trying to open module; module='afsocket', filename='/usr/local/lib/syslog-ng/libafsocket.so' Trying to open module; module='afuser', filename='/usr/local/lib/syslog-ng/libafuser.so' Trying to open module; module='basicfuncs', filename='/usr/local/lib/syslog-ng/libbasicfuncs.so' Trying to open module; module='csvparser', filename='/usr/local/lib/syslog-ng/libcsvparser.so' Trying to open module; module='dbparser', filename='/usr/local/lib/syslog-ng/libdbparser.so' Trying to open module; module='syslogformat', filename='/usr/local/lib/syslog-ng/libsyslogformat.so' Trying to open module; module='afsql', filename='/usr/local/lib/syslog-ng/libafsql.so' Syslog connection established; fd='8', server='AF_INET(192.168.0.7:514)', local='AF_INET(0.0.0.0:0)' Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.3.3' Incoming log entry; line='<189>41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)' Incoming log entry; line='<189>Dec 9 08:41:24 6500-1 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' Incoming log entry; line='<189>Dec 9 08:41:24 localhost 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' Incoming log entry; line='<189>Dec 9 08:41:24 localhost 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' Incoming log entry; line='<189>Dec 9 08:41:24 localhost 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' ....forever....
Sounds like messages sent to 192.168.0.7 are feeded back to syslog-ng so there is a logging loop. Is this address local? When not then there is a chance that the packet filter rule isn't correct. On Fri, Dec 9, 2011 at 10:34 AM, Dave Haywood <tla@oak.selfip.net> wrote:
Hi,
I have a problem with syslog-ng 3.3.3. When I have forwarding enabled to a remote syslog server (via UDP) syslog-ng repeatedly writes the same message(s) to the log file and only stops when the disk is full. Using tcpdump on the remote server, I don't see any data arrive from the syslog-ng server so forwarding is not working either.
When I remove the forwarding part of the config file the local file is written correctly (ie once). If I remove the local file part from the config file and only enable the forwarding, I see syslog-ng take all the CPU time. I never see any syslog messages arrive at the remote syslog server.
I tried: 1) disabling IPv6 - no change 2) running outside the chroot jail - no change 3) running as userid root - no change
Does anyone have any idea what would cause this? Debug info below.
The environment is:
RedHat AS 4.8 (linux 2.6.9-89.ELsmp) on vmware ESXi 4.1.0
All required software built and installed in /usr/local/ :
eventlog_0.2.12.tar.gz gettext-0.18.1.1.tar.gz glib-2.29.90.tar.bz2 libdbi-0.8.4.tar.gz libdbi-drivers-0.8.3.tar.gz libffi-3.0.9.tar.gz libnet-0.10.11.tar.gz pkg-config-0.26.tar.gz Python-2.7.2.tar.bz2 zlib-1.2.5.tar.bz2 syslog-ng_3.3.3.tar.gz
syslog-ng is running chroot() in directory /data as user syslogng:sysadmins and listens on port 1514. iptables redirects any incoming port 514 traffic to 1514. The required /usr/local/ directories are mounted (-o bind) under /data.
syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Dec 8 2011 17:46:40 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: convertfuncs,affile,afmongodb,dummy,basicfuncs,csvparser,confgen,afsql,syslogformat,afuser,afsocket,afprog,afsocket-notls,dbparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: off
Config file:
@version: 3.3
source s_udp { udp(ip("0.0.0.0") port(1514)); };
destination file1 { file("/log/network.log" owner(syslogng)group(sysops) perm(0640) flags(no-multi-line)); };
destination NeDi { udp("192.168.0.7" port(514)); };
log { source(s_udp); destination(file1); };
# enabling the line below breaks logging to the file above
log { source(s_udp); destination(NeDi); };
Debug:
# /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng.conf --chroot=/data --user=syslogng --group=sysadmins --persist-file=/log/syslog-ng.persist --foreground --process-mode=foreground --stderr --debug nanosleep() is not accurate enough to introduce minor stalls on the reader side, multi-threaded performance may be affected; Trying to open module; module='affile', filename='/usr/local/lib/syslog-ng/libaffile.so' Trying to open module; module='afprog', filename='/usr/local/lib/syslog-ng/libafprog.so' Trying to open module; module='afsocket', filename='/usr/local/lib/syslog-ng/libafsocket.so' Trying to open module; module='afuser', filename='/usr/local/lib/syslog-ng/libafuser.so' Trying to open module; module='basicfuncs', filename='/usr/local/lib/syslog-ng/libbasicfuncs.so' Trying to open module; module='csvparser', filename='/usr/local/lib/syslog-ng/libcsvparser.so' Trying to open module; module='dbparser', filename='/usr/local/lib/syslog-ng/libdbparser.so' Trying to open module; module='syslogformat', filename='/usr/local/lib/syslog-ng/libsyslogformat.so' Trying to open module; module='afsql', filename='/usr/local/lib/syslog-ng/libafsql.so' Syslog connection established; fd='8', server='AF_INET(192.168.0.7:514)', local='AF_INET(0.0.0.0:0)' Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.3.3' Incoming log entry; line='<189>41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)' Incoming log entry; line='<189>Dec 9 08:41:24 6500-1 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' Incoming log entry; line='<189>Dec 9 08:41:24 localhost 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' Incoming log entry; line='<189>Dec 9 08:41:24 localhost 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' Incoming log entry; line='<189>Dec 9 08:41:24 localhost 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' ....forever....
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On 09/12/2011 09:53, Sandor Geller wrote:
Sounds like messages sent to 192.168.0.7 are feeded back to syslog-ng so there is a logging loop. Is this address local? When not then there is a chance that the packet filter rule isn't correct. Thanks! You were right, the issue was with the iptables rule. I was trying to capture traffic from localhost to port 514 and redirect it to 1514 using NAT table OUTPUT. I use this for testing every facility / severity combination during install. But I didn't specify a destination host (of the local IP address); I only specified the port. This meant and traffic forwarded to a remote host is redirected by iptables back to the localhost, causing a loop.
Thanks for the help :)
On Fri, Dec 9, 2011 at 10:34 AM, Dave Haywood <tla@oak.selfip.net> wrote:
Hi,
I have a problem with syslog-ng 3.3.3. When I have forwarding enabled to a remote syslog server (via UDP) syslog-ng repeatedly writes the same message(s) to the log file and only stops when the disk is full. Using tcpdump on the remote server, I don't see any data arrive from the syslog-ng server so forwarding is not working either.
When I remove the forwarding part of the config file the local file is written correctly (ie once). If I remove the local file part from the config file and only enable the forwarding, I see syslog-ng take all the CPU time. I never see any syslog messages arrive at the remote syslog server.
I tried: 1) disabling IPv6 - no change 2) running outside the chroot jail - no change 3) running as userid root - no change
Does anyone have any idea what would cause this? Debug info below.
The environment is:
RedHat AS 4.8 (linux 2.6.9-89.ELsmp) on vmware ESXi 4.1.0
All required software built and installed in /usr/local/ :
eventlog_0.2.12.tar.gz gettext-0.18.1.1.tar.gz glib-2.29.90.tar.bz2 libdbi-0.8.4.tar.gz libdbi-drivers-0.8.3.tar.gz libffi-3.0.9.tar.gz libnet-0.10.11.tar.gz pkg-config-0.26.tar.gz Python-2.7.2.tar.bz2 zlib-1.2.5.tar.bz2 syslog-ng_3.3.3.tar.gz
syslog-ng is running chroot() in directory /data as user syslogng:sysadmins and listens on port 1514. iptables redirects any incoming port 514 traffic to 1514. The required /usr/local/ directories are mounted (-o bind) under /data.
syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Dec 8 2011 17:46:40 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Available-Modules: convertfuncs,affile,afmongodb,dummy,basicfuncs,csvparser,confgen,afsql,syslogformat,afuser,afsocket,afprog,afsocket-notls,dbparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: off Enable-Pcre: off
Config file:
@version: 3.3
source s_udp { udp(ip("0.0.0.0") port(1514)); };
destination file1 { file("/log/network.log" owner(syslogng)group(sysops) perm(0640) flags(no-multi-line)); };
destination NeDi { udp("192.168.0.7" port(514)); };
log { source(s_udp); destination(file1); };
# enabling the line below breaks logging to the file above
log { source(s_udp); destination(NeDi); };
Debug:
# /usr/local/sbin/syslog-ng --cfgfile=/usr/local/etc/syslog-ng.conf --chroot=/data --user=syslogng --group=sysadmins --persist-file=/log/syslog-ng.persist --foreground --process-mode=foreground --stderr --debug nanosleep() is not accurate enough to introduce minor stalls on the reader side, multi-threaded performance may be affected; Trying to open module; module='affile', filename='/usr/local/lib/syslog-ng/libaffile.so' Trying to open module; module='afprog', filename='/usr/local/lib/syslog-ng/libafprog.so' Trying to open module; module='afsocket', filename='/usr/local/lib/syslog-ng/libafsocket.so' Trying to open module; module='afuser', filename='/usr/local/lib/syslog-ng/libafuser.so' Trying to open module; module='basicfuncs', filename='/usr/local/lib/syslog-ng/libbasicfuncs.so' Trying to open module; module='csvparser', filename='/usr/local/lib/syslog-ng/libcsvparser.so' Trying to open module; module='dbparser', filename='/usr/local/lib/syslog-ng/libdbparser.so' Trying to open module; module='syslogformat', filename='/usr/local/lib/syslog-ng/libsyslogformat.so' Trying to open module; module='afsql', filename='/usr/local/lib/syslog-ng/libafsql.so' Syslog connection established; fd='8', server='AF_INET(192.168.0.7:514)', local='AF_INET(0.0.0.0:0)' Running application hooks; hook='1' Running application hooks; hook='3' syslog-ng starting up; version='3.3.3' Incoming log entry; line='<189>41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)' Incoming log entry; line='<189>Dec 9 08:41:24 6500-1 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' Incoming log entry; line='<189>Dec 9 08:41:24 localhost 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' Incoming log entry; line='<189>Dec 9 08:41:24 localhost 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' Incoming log entry; line='<189>Dec 9 08:41:24 localhost 41609: Dec 9 2011 08:41:22.691 UTC: %SYS-5-CONFIG_I: Configured from console by admin on vty0 (FD96:D107:BCF2::3)\x0a' ....forever....
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Dave Haywood
-
Sandor Geller