[syslog-ng] 1.6.5 performance

[Scott C]

The log data we receive from most systems is fairly consistent from day to day.  That's how I can tell I'm missing some data.  But it's not to the extent that syslog-ng STATS would seem to indicate.
 
As for the system-level performance stuff, I don't mean to sound pretentious, but I've effectively ruled all of that out.  I've analyzed all the statistics, and I even opened a ticket with Sun to get their two cents worth.  They came to the same conclusion that I did--the application is simply not keeping up with the IP stack.
 
Also, although it may not be entirely clear in my original post, these relays are just relays.  The log messages come in via udp or tcp and go right back out via tcp.  Any and all processing (including dns and regexp) is performed at the back end.
 
At the end of the day, I guess my real question is, how truly accurate is the STATS number that I'm seeing?
 
Here is my syslog-ng.conf:
 
options { stats(300); sync(0); time_reopen(10); log_fifo_size(3000000); gc_busy_threshold(200000); check_hostname(no); keep_hostname(yes); };
# Grab syslog-ng specific messages.
source ng_local {
        internal();
};
# Grab local messages (OS specific messages).
source local {
        sun-streams("/dev/log" door("/etc/.syslog_door"));
};
source remote {
        udp(ip("1.1.1.1") port(514));
        tcp(ip("1.1.1.1") port(5514) max-connections(50));
};
# Log location for syslog-ng specific messages.
destination ng_messages {
        file("/var/log/syslog-ng" owner(root) group(staff) perm(0740) create_dirs(yes) dir_perm(0755));
};
# Log location for OS specific messages.
destination local_messages {
        file("/var/adm/messages" owner(root) group(staff) create_dirs(yes) dir_perm(0755));
};
# Forward messages to the cluster
destination remote_messages {
        tcp("1.1.1.50" port(5514));
};
# Log all syslog-ng specific messages.
log { source(ng_local); destination(ng_messages); };
# Log all OS specific messages.
log { source(local); destination(local_messages); };
# Log all remote system messages.
log { source(remote); destination(remote_messages); };

Nate Campi <nate at campin.net> wrote:
On Fri, Oct 21, 2005 at 06:54:04AM -0700, Scott C wrote:
> But what's really most peculiar in this scenario is the fact that the
> numbers simply don't add up. Why does syslog-ng appear (on the
> surface) to be dropping a very large percentage of the messages that
> it receives? I realize that it's not, but the numbers tell a
> different story. And how could it possibly drop so many messages when
> the FIFO queue is configured to buffer three million lines? How
> preposterous!

So you think you really have all the logs but you see STATS messages
reporting dropped messages? How would you know if you really have them
all? It's possible that under heavy load you have some program or pipe
destination (or maybe even file if you have slow disks) that just can't
keep up. That's not syslog-ng's fault, it just lets you know that the
buffer filled up.

Right now all anyone can do is shoot off wild guesses like mine above,
since there's no hard data in your post, just your conclusions. If you
want to post your syslog-ng.conf, output of system commands like
"netstat -i", prstat, "iostat -mnPxz 10" and vmstat during peak loads,
and whatever else you used to reach your conclusions then we'd be in a
better position to help.

OBTW there are performance tips in the FAQ that give clues as to causes:

http://www.campin.net/syslog-ng/faq.html#perf

Possible culprits: DNS, regexps (though you say CPU is ok, so maybe
not), logging to a tty or the console.
-- 
Nate

"I must've seen it in a USENET posting; that's sort of like hearsay
evidence from Richard Nixon..." - Houghton, Blair

_______________________________________________
syslog-ng maillist - syslog-ng at lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html




		
---------------------------------
 Yahoo! FareChase - Search multiple travel sites in one click.  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20051021/1c027ba8/attachment.html