[syslog-ng] syslog-ng performance tuning

[Rémi BUISSON]

Hi everybody,

I'have an issue with syslog-ng configuration.
I would like to centralize my logs on one server.

I've a lot of logs to send. I don't know how many but I can estimate it 
to 500GB per day from decades of servers.
But, it writes only 25 GB per day.
For some reasons I work on a debian etchnhalf environnement.
So, I'm working with syslog-ng 2.0.0.

I wrote a perl program which spawn p "logger -p local5.info" processes
and send n lines of m characters.

I'have tested with:
p: 1 000
n: 1 000
m: 1 000

Instead of having 1 000 000 lines in my logs I have nearly 10 000 lines !
But my test was not revelant because normal logs where not stopped. So, 
maybe normal.

I've googled my issue and tried to play on log_fifo_size(),
max_connections() and sync() on both server and client but nothing to do ...

Here is my client configuration file:

options {
        chain_hostnames(0);

        time_reopen(10);

        time_reap(360);
        log_fifo_size(2048);

        create_dirs(yes);

        group(adm);
        perm(0640);

        dir_perm(0755);

        use_dns(no);
        stats_freq(0);

        bad_hostname("^gconfd$");
};
source s_all {
        internal();
        unix-stream("/dev/log");
        file("/proc/kmsg" log_prefix("kernel: "));
};

destination dn_fslog {
        tcp("xxx.xxx.xxx.xxx" port(514));
};

filter f_local5 {
         facility(local5);
};

log {
        source(s_all);
        filter(f_local5);
        destination(dn_fslog);
};

# for all my apache vhosts
destination df_access_www.foo.com {
file("/var/www/www.foo.com/logs/access.log"); };

filter f_local0_access_www.foo.com {
        level(info)
        and facility(local0)
        and program("www\.foo\.com");
};

log {
        source(s_all);
        filter(f_local0_access_www.foo.com);
        destination(df_access_www.foo.com);
        destination(dn_fslog);
};
destination df_error_www.foo.com {
file("/var/www/www.foo.com/logs/error.log"); };

filter f_local0_error_www.foo.com {
        level(error)
        and facility(local0)
        and program("www\.foo\.com");
};

log {
        source(s_all);
        filter(f_local0_error_www.foo.com);
        destination(df_error_www.foo.com);
        destination(dn_fslog);
};

etc with www.foo2.com

Here is my server configuration file:
options {
        chain_hostnames(0);

        time_reopen(10);

        time_reap(360);
        log_fifo_size(2048);

        create_dirs(yes);

        group(adm);
        perm(0640);

        dir_perm(0755);

        use_dns(no);
        stats_freq(0);

        bad_hostname("^gconfd$");
};

source s_net {
        tcp(ip(xxx.xxx.xxx.xxx) port(514));
};

destination df_net_access_aaa.aaa.aaa.aaa-www.foo.com {
file("/logs/www.foo.com/aaa.aaa.aaa.aaa-access.log"); };

filter f_local0_access_aaa.aaa.aaa.aaa-www.foo.com {
        level(info)
        and facility(local0)
        and program("www\.foo\.com")
        and host("aaa\.aaa\.aaa\.aaa");
};

log {
        source(s_net);
        filter(f_local0_access_aaa.aaa.aaa.aaa-www.foo.com);
        destination(df_net_access_aaa.aaa.aaa.aaa-www.foo.com);
};
destination df_net_error_aaa.aaa.aaa.aaa-www.foo.com {
file("/logs/www.foo.com/aaa.aaa.aaa.aaa-error.log"); };

filter f_local0_error_aaa.aaa.aaa.aaa-www.foo.com {
        level(error)
        and facility(local0)
        and program("www\.foo\.com")
        and host("aaa\.aaa\.aaa\.aaa");
};

log {
        source(s_net);
        filter(f_local0_error_aaa.aaa.aaa.aaa-www.foo.com);
        destination(df_net_error_aaa.aaa.aaa.aaa-www.foo.com);
};

etc with aaa.aaa.aaa.bbb and www.foo2.com


Just to give you a number I'm speaking about 20 web servers with 20 load
balanced websites on each.
Servers are on a Gigabit networks with bonded Gigabit interfaces.
It is running on Linux 2.6.24-etchnhalf.1-686-bigmem.

Maybe syslog-ng cannot handle 400 different filter rules but it would
suprize me. If it's the case, in there any better way to separate, for 
instance, apache error and access log for in each server, in a file name 
/logs/<project = progname>/<ip>-access.log and /logs/<project = 
progname>/<ip>-error.log ?

I don't think my hardware is in cause because cpu average 98% idle and 
no io wait.

Can you tell me what I misunderstood or simply what's wrong in my
configuration files.
Or, it may work better with a 3.x version compiled with some options ?

Thanks in advance.

Regards,

Rémi