[syslog-ng]Missing Log Entries under Load

[Tim Burress]

Hello!

Apparently this is sort of a FAQ, but despite wading
through many Google searches, I haven't really found a
good explanation or set of suggestions, so I thought I
would post the question and then duck.

We're running:

    syslog-ng    1.6.4
    iptables     v1.2.11-20040621
    Linux        2.4.26

with iptables rules set up to log incoming
connections. What we see is that, when we do a very
fast port scan, the logs are very incomplete. For
instance, in a scan of 440 ports, only about 210
entries appear in the logs. We also get a small number
of corrupted log entries in our default messages file,
where leading characters of the message appear to have
been lost.

We have our log sources set up as:

source src {
        internal();
        file("/proc/kmsg");
        unix-stream ("/dev/log" max-connections(200));
        unix-stream ("/var/log/snort/dev/log"
max-connections(30));
};

Could someone explain, or point me to an explanation
of, why these problems occur? It seems likely that the
message corruption problem is due to wrap-around of
the kernel message ring buffer. I suppose the missing
messages could also be caused by that, however
doubling the size of the buffer and rebuilding the
kernel didn't seem to have any effect.

And then, are there any suggestions for ways to tune
the system so that a greater proportion of messages
can be logged? We've tried tweaking the syslog-ng FIFO
size and garbage collection parameters, but these,
too, seemed to have little effect, at least in
isolation.

Any suggestions greatly appreciated!

Thanks!

Tim


	
		
__________________________________
Do you Yahoo!?
New and Improved Yahoo! Mail - 100MB free storage!
http://promotions.yahoo.com/new_mail