[syslog-ng] Losing to much remote sent logs

[Martin Holste]

If possible, I would try swapping the $HOST macro for $SOURCEIP to
avoid doing any DNS lookups, cached or not.  It's unlikely to help,
but it sounds like you've already tried the basic tuning things.  I
will say that I'm very surprised you're losing log lines.  What is
your peak logs per second, and how long are the peaks?

On Fri, Mar 2, 2012 at 3:40 AM, Daniel Neubacher
<daniel.neubacher at xing.com> wrote:
> Hello there,
>
> I’ve started playing around with syslog-ng 3.3.4 ose a few days ago but I’m
> still experiencing some trouble. First of all we want to use syslog-ng to
> send all of our logs via udp to a central syslog server. This includes of
> course syslogs, apache logs and custom generated applogs. These logs are
> generated from 400 clients and produces a minimum of 300 mio. log lines a
> day.
>
> The problem is really simple: I’m losing log lines :P Most of the time
> everything goes well but when the logs are peaking high 1-5% logs are
> getting lost.
>
> Last night the stats of the server and a client said 0 drops but when I
> counted the lines I found lost lines. The server has 24g ram & 8 cores and I
> can rule out a network problem for sure.
>
>
>
> So now to my questions, has anyone else an idea where I can tweak my cfg or
> where I have to look to find more clues? Is tcp the only way to get around
> it?
>
> I’ve attached my syslog server cfg. The so_rcvbuf buffer is the same size as
> the os net.core.rmem settings. And as described in the various balabit blog
> posts I played around with log_fetch_limit and flush_lines already.
>
>
>
> syslog-ng.conf:
>
> @version: 3.3
>
>
>
> options {
>
>     threaded(yes);
>
>     owner("root");
>
>     group("root");
>
>     perm(0660);
>
>
>
>     dir_owner("root");
>
>     dir_group("root");
>
>     dir_perm(0770);
>
>     create_dirs(yes);
>
>
>
>     stats_freq(600);
>
>     stats_level(2);
>
>     chain_hostnames(yes);
>
>     normalize_hostnames(yes);
>
>     check_hostname(yes);
>
>
>
>     dns_cache(yes);
>
>     dns_cache_size(16384);
>
>     dns_cache_expire(3600);
>
>     dns_cache_expire_failed(60);
>
>
>
>     log_msg_size(16384);
>
>     log_fifo_size(100000);
>
>
>
>
>
>     use_fqdn(yes);
>
> #disabled 4 debugging
>
> #    flush_lines(200);
>
> };
>
>
>
> source s_src {
>
>         unix-dgram("/dev/log");
>
>         internal();
>
>         file("/proc/kmsg" program_override("kernel"));
>
> };
>
>
>
> source s_net {
>
> udp(
>
>         log_fetch_limit(400)
>
>         so_rcvbuf(51200000)
>
>         keep_hostname(yes)
>
>         keep_timestamp(no)
>
>         ip("10.8.4.10")
>
>         port(514)
>
> );
>
> tcp(
>
>         so_rcvbuf(51200000)
>
>         so_keepalive(yes)
>
>         keep_hostname(no)
>
>         keep_timestamp(no)
>
>         ip("10.8.4.10")
>
>         port(514)
>
>
>
> );
>
> syslog();
>
> };
>
>
>
> filter f_syslog {
>
>      not program(access.log) and
>
>      not program(error.log) and
>
>      not program(beetle.log) and
>
>      not program(edge.log);
>
>
>
> };
>
>
>
> filter f_apache {
>
>     program(access.log) or
>
>     program(error.log);
>
> };
>
>
>
> filter f_applogs {
>
>     program(beetle.log)
>
>     or program(edge.log);
>
> };
>
>
>
> template t_plain {
>
>     template("$MSG\n"); template_escape(no);
>
> };
>
>
>
> destination d_messages { file("/var/log/messages"); };
>
> destination d_remote {
> file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); };
>
> destination d_apache {
> file("/log/apache/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
> destination d_applogs {
> file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
> template(t_plain)); };
>
>
>
> log {
>
>     source(s_src);
>
>     destination(d_messages);
>
> };
>
>
>
> log {
>
>     source(s_net);
>
>     filter(f_syslog);
>
>     destination(d_remote);
>
> };
>
>
>
> log {
>
>     source(s_net);
>
>     filter(f_apache);
>
>     destination(d_apache);
>
> };
>
>
>
> log {
>
>     source(s_net);
>
>     filter(f_applogs);
>
>     destination(d_applogs);
>
> };
>
>
>
>
>
> Thanks
>
> Daniel Neubacher
>
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>