Given that you have only file() destinations, performance should not be an issue, so something is definitely wrong. One other shot in the dark: is your log server a VM, and if so, is there any chance that its not getting enough resources because of the extra mail traffic? On Mon, Mar 5, 2012 at 4:30 AM, Daniel Neubacher <daniel.neubacher@xing.com> wrote:
Thanks for the answer. Disabling DNS would be really painful. I will play around some more today and try it as a last resort. The Baseline for a webserver is 146k logs per hour, the minimum is 22k and the maximum 365k. The peaks are only happening in the night for 3-4 hours because of the local mail traffic. Today I will roll out my tcp logging conf but I'm not too happy about that.
-----Ursprüngliche Nachricht----- Von: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] Im Auftrag von Martin Holste Gesendet: Freitag, 2. März 2012 16:00 An: Syslog-ng users' and developers' mailing list Betreff: Re: [syslog-ng] Losing to much remote sent logs
If possible, I would try swapping the $HOST macro for $SOURCEIP to avoid doing any DNS lookups, cached or not. It's unlikely to help, but it sounds like you've already tried the basic tuning things. I will say that I'm very surprised you're losing log lines. What is your peak logs per second, and how long are the peaks?
On Fri, Mar 2, 2012 at 3:40 AM, Daniel Neubacher <daniel.neubacher@xing.com> wrote:
Hello there,
I've started playing around with syslog-ng 3.3.4 ose a few days ago but I'm still experiencing some trouble. First of all we want to use syslog-ng to send all of our logs via udp to a central syslog server. This includes of course syslogs, apache logs and custom generated applogs. These logs are generated from 400 clients and produces a minimum of 300 mio. log lines a day.
The problem is really simple: I'm losing log lines :P Most of the time everything goes well but when the logs are peaking high 1-5% logs are getting lost.
Last night the stats of the server and a client said 0 drops but when I counted the lines I found lost lines. The server has 24g ram & 8 cores and I can rule out a network problem for sure.
So now to my questions, has anyone else an idea where I can tweak my cfg or where I have to look to find more clues? Is tcp the only way to get around it?
I've attached my syslog server cfg. The so_rcvbuf buffer is the same size as the os net.core.rmem settings. And as described in the various balabit blog posts I played around with log_fetch_limit and flush_lines already.
syslog-ng.conf:
@version: 3.3
options {
threaded(yes);
owner("root");
group("root");
perm(0660);
dir_owner("root");
dir_group("root");
dir_perm(0770);
create_dirs(yes);
stats_freq(600);
stats_level(2);
chain_hostnames(yes);
normalize_hostnames(yes);
check_hostname(yes);
dns_cache(yes);
dns_cache_size(16384);
dns_cache_expire(3600);
dns_cache_expire_failed(60);
log_msg_size(16384);
log_fifo_size(100000);
use_fqdn(yes);
#disabled 4 debugging
# flush_lines(200);
};
source s_src {
unix-dgram("/dev/log");
internal();
file("/proc/kmsg" program_override("kernel"));
};
source s_net {
udp(
log_fetch_limit(400)
so_rcvbuf(51200000)
keep_hostname(yes)
keep_timestamp(no)
ip("10.8.4.10")
port(514)
);
tcp(
so_rcvbuf(51200000)
so_keepalive(yes)
keep_hostname(no)
keep_timestamp(no)
ip("10.8.4.10")
port(514)
);
syslog();
};
filter f_syslog {
not program(access.log) and
not program(error.log) and
not program(beetle.log) and
not program(edge.log);
};
filter f_apache {
program(access.log) or
program(error.log);
};
filter f_applogs {
program(beetle.log)
or program(edge.log);
};
template t_plain {
template("$MSG\n"); template_escape(no);
};
destination d_messages { file("/var/log/messages"); };
destination d_remote { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); };
destination d_apache { file("/log/apache/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); };
destination d_applogs { file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); };
log {
source(s_src);
destination(d_messages);
};
log {
source(s_net);
filter(f_syslog);
destination(d_remote);
};
log {
source(s_net);
filter(f_apache);
destination(d_apache);
};
log {
source(s_net);
filter(f_applogs);
destination(d_applogs);
};
Thanks
Daniel Neubacher
______________________________________________________________________ ________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq