[syslog-ng] Losing to much remote sent logs

[Daniel Neubacher]

Thanks for the answer. Disabling DNS would be really painful. I will play around some more today and try it as a last resort. 
The Baseline for a webserver is 146k logs per hour, the minimum is 22k and the maximum 365k. The peaks are only happening in the night for 3-4 hours because of the local mail traffic.
Today I will roll out my tcp logging conf but I'm not too happy about that. 


-----Ursprüngliche Nachricht-----
Von: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] Im Auftrag von Martin Holste
Gesendet: Freitag, 2. März 2012 16:00
An: Syslog-ng users' and developers' mailing list
Betreff: Re: [syslog-ng] Losing to much remote sent logs

If possible, I would try swapping the $HOST macro for $SOURCEIP to avoid doing any DNS lookups, cached or not.  It's unlikely to help, but it sounds like you've already tried the basic tuning things.  I will say that I'm very surprised you're losing log lines.  What is your peak logs per second, and how long are the peaks?

On Fri, Mar 2, 2012 at 3:40 AM, Daniel Neubacher <daniel.neubacher at xing.com> wrote:
> Hello there,
>
> I've started playing around with syslog-ng 3.3.4 ose a few days ago 
> but I'm still experiencing some trouble. First of all we want to use 
> syslog-ng to send all of our logs via udp to a central syslog server. 
> This includes of course syslogs, apache logs and custom generated 
> applogs. These logs are generated from 400 clients and produces a 
> minimum of 300 mio. log lines a day.
>
> The problem is really simple: I'm losing log lines :P Most of the time 
> everything goes well but when the logs are peaking high 1-5% logs are 
> getting lost.
>
> Last night the stats of the server and a client said 0 drops but when 
> I counted the lines I found lost lines. The server has 24g ram & 8 
> cores and I can rule out a network problem for sure.
>
>
>
> So now to my questions, has anyone else an idea where I can tweak my 
> cfg or where I have to look to find more clues? Is tcp the only way to 
> get around it?
>
> I've attached my syslog server cfg. The so_rcvbuf buffer is the same 
> size as the os net.core.rmem settings. And as described in the various 
> balabit blog posts I played around with log_fetch_limit and flush_lines already.
>
>
>
> syslog-ng.conf:
>
> @version: 3.3
>
>
>
> options {
>
>     threaded(yes);
>
>     owner("root");
>
>     group("root");
>
>     perm(0660);
>
>
>
>     dir_owner("root");
>
>     dir_group("root");
>
>     dir_perm(0770);
>
>     create_dirs(yes);
>
>
>
>     stats_freq(600);
>
>     stats_level(2);
>
>     chain_hostnames(yes);
>
>     normalize_hostnames(yes);
>
>     check_hostname(yes);
>
>
>
>     dns_cache(yes);
>
>     dns_cache_size(16384);
>
>     dns_cache_expire(3600);
>
>     dns_cache_expire_failed(60);
>
>
>
>     log_msg_size(16384);
>
>     log_fifo_size(100000);
>
>
>
>
>
>     use_fqdn(yes);
>
> #disabled 4 debugging
>
> #    flush_lines(200);
>
> };
>
>
>
> source s_src {
>
>         unix-dgram("/dev/log");
>
>         internal();
>
>         file("/proc/kmsg" program_override("kernel"));
>
> };
>
>
>
> source s_net {
>
> udp(
>
>         log_fetch_limit(400)
>
>         so_rcvbuf(51200000)
>
>         keep_hostname(yes)
>
>         keep_timestamp(no)
>
>         ip("10.8.4.10")
>
>         port(514)
>
> );
>
> tcp(
>
>         so_rcvbuf(51200000)
>
>         so_keepalive(yes)
>
>         keep_hostname(no)
>
>         keep_timestamp(no)
>
>         ip("10.8.4.10")
>
>         port(514)
>
>
>
> );
>
> syslog();
>
> };
>
>
>
> filter f_syslog {
>
>      not program(access.log) and
>
>      not program(error.log) and
>
>      not program(beetle.log) and
>
>      not program(edge.log);
>
>
>
> };
>
>
>
> filter f_apache {
>
>     program(access.log) or
>
>     program(error.log);
>
> };
>
>
>
> filter f_applogs {
>
>     program(beetle.log)
>
>     or program(edge.log);
>
> };
>
>
>
> template t_plain {
>
>     template("$MSG\n"); template_escape(no);
>
> };
>
>
>
> destination d_messages { file("/var/log/messages"); };
>
> destination d_remote {
> file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); };
>
> destination d_apache {
> file("/log/apache/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
> destination d_applogs {
> file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
>
>
> log {
>
>     source(s_src);
>
>     destination(d_messages);
>
> };
>
>
>
> log {
>
>     source(s_net);
>
>     filter(f_syslog);
>
>     destination(d_remote);
>
> };
>
>
>
> log {
>
>     source(s_net);
>
>     filter(f_apache);
>
>     destination(d_apache);
>
> };
>
>
>
> log {
>
>     source(s_net);
>
>     filter(f_applogs);
>
>     destination(d_applogs);
>
> };
>
>
>
>
>
> Thanks
>
> Daniel Neubacher
>
>
>
>
> ______________________________________________________________________
> ________ Member info: 
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq