[syslog-ng] syslog-ng v3.0.4 stops receiving remote logs

[Mike Duncan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

We are running into an issue using syslog-ng v3.0.3 and v3.0.4. After
about 1minute or so, all external/remote hosts can no longer log to the
logging server (syslog-ng). We run udp only and have something around
300-400 hosts logging to the server. The local logging seems to be okay
even after the remote logging ceases. We are not getting any indication
from syslog-ng (logs or stdout) that there is an issue unless we run it
in debug mode.

Firstly, we tried to ensure the rmem and wmem were set to handle
everything. We were seeing some 30million packet receive errors, so
bumped up the settings from 8MB to 16MB. This did not help resolve the
issue.

Secondly, after looking through mailing lists and various posts online,
we thought the issue to be the 0byte-udp-packet bug as mentioned here:
https://bugzilla.balabit.com/show_bug.cgi?id=32. When running v3.0.3 in
debug mode with trace enabled, we saw the EOF error messages which lead
us to download v3.0.4, apply the patch and compile. Unfortunately, we
are still having the same issue as before but no EOF error messages this
time.

ulimit is set to unlimited and at most, we only get 30-40 fd's opened.
So, we are at a loss and hoping to get some input from here. Here are
some details...(sorry for wrapping issues)...

netstat -su snippet
===========================
Udp:
    259879531 packets received
    709190 packets to unknown port received.
    29579106 packet receive errors
    328899408 packets sent
    RcvbufErrors: 29579106

netstat -anp snippet
===========================
udp   16776712      0 0.0.0.0:514             0.0.0.0:*
          -


sysctl snippet
===========================
net.core.rmem_default = 16777216
net.core.rmem_max = 16777216
net.core.wmem_default = 16777216
net.core.wmem_max = 16777216

syslog-ng.conf snippets
===========================
options {
      chain_hostnames(0);
      time_reopen(10);
      time_reap(360);
      log_fifo_size(2048);
      create_dirs(yes);
      group(adm);
      perm(0640);
      dir_perm(0755);
      use_dns(yes);
      #log_msg_size(2048);
      stats_freq(0);
      bad_hostname("^gconfd$");
};

source s_all {
        internal();
        unix-stream("/dev/log");
        file("/proc/kmsg" program_override("kernel: "));
        udp(ip(0.0.0.0) port(514));
};

#example remote logging
destination df_windows { file("/var/log/windows.log"); };
filter f_windows { program("MSWinEventLog"); };
log { source(s_all); filter(f_windows); destination(df_windows);
flags(final); };

#another example
destination df_networkdevice { file("/var/log/networkdevice.log"); };
filter f_2e { host("2e[0-9]") or host("2eg[0-9]"); }; #2nd floor East
log { source(s_all); filter(f_2e); destination(df_networkdevice);
flags(final); };

uname -a
=============================
Linux logalot 2.6.24-24-server #1 SMP Fri Jul 24 23:27:55 UTC 2009 i686
GNU/Linux

ps ax | grep syslog
=============================
 2201 ?        S      0:00 supervising syslog-ng
 2202 ?        Ss     0:00 /opt/syslog-ng/sbin/syslog-ng --no-caps


Are we running into a limit of some sort or a bug or more likely -- are
we missing something? Any help is appreciated. If you need more
information, just let me know.

Thanks.

- --
Mike Duncan
ISSO, Application Security Specialist
Government Contractor with STG, Inc.
NOAA :: National Climatic Data Center

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkqzyA4ACgkQnvIkv6fg9hZdiQCgkO6wW8yo9AC49f7E+sEQxD7J
nH4AmwYNG/zsLJugvy+apqrueG+UR4TS
=Y6Ol
-----END PGP SIGNATURE-----