[syslog-ng] Losing to much remote sent logs

[Daniel Neubacher]

No it's not a vm. But tcp syslog is performing pretty good. I just wonder what happens if the syslog server isn't reachable for a long time. I tested it with another syslog server and it went rogue.

Today it didn't lose a line so tcp is pretty much the answer :P

-----Ursprüngliche Nachricht-----
Von: syslog-ng-bounces at lists.balabit.hu [mailto:syslog-ng-bounces at lists.balabit.hu] Im Auftrag von Martin Holste
Gesendet: Montag, 5. März 2012 17:59
An: Syslog-ng users' and developers' mailing list
Betreff: Re: [syslog-ng] Losing to much remote sent logs

Given that you have only file() destinations, performance should not be an issue, so something is definitely wrong.  One other shot in the
dark: is your log server a VM, and if so, is there any chance that its not getting enough resources because of the extra mail traffic?

On Mon, Mar 5, 2012 at 4:30 AM, Daniel Neubacher <daniel.neubacher at xing.com> wrote:
> Thanks for the answer. Disabling DNS would be really painful. I will play around some more today and try it as a last resort.
> The Baseline for a webserver is 146k logs per hour, the minimum is 22k and the maximum 365k. The peaks are only happening in the night for 3-4 hours because of the local mail traffic.
> Today I will roll out my tcp logging conf but I'm not too happy about that.
>
>
> -----Ursprüngliche Nachricht-----
> Von: syslog-ng-bounces at lists.balabit.hu 
> [mailto:syslog-ng-bounces at lists.balabit.hu] Im Auftrag von Martin 
> Holste
> Gesendet: Freitag, 2. März 2012 16:00
> An: Syslog-ng users' and developers' mailing list
> Betreff: Re: [syslog-ng] Losing to much remote sent logs
>
> If possible, I would try swapping the $HOST macro for $SOURCEIP to avoid doing any DNS lookups, cached or not.  It's unlikely to help, but it sounds like you've already tried the basic tuning things.  I will say that I'm very surprised you're losing log lines.  What is your peak logs per second, and how long are the peaks?
>
> On Fri, Mar 2, 2012 at 3:40 AM, Daniel Neubacher <daniel.neubacher at xing.com> wrote:
>> Hello there,
>>
>> I've started playing around with syslog-ng 3.3.4 ose a few days ago 
>> but I'm still experiencing some trouble. First of all we want to use 
>> syslog-ng to send all of our logs via udp to a central syslog server.
>> This includes of course syslogs, apache logs and custom generated 
>> applogs. These logs are generated from 400 clients and produces a 
>> minimum of 300 mio. log lines a day.
>>
>> The problem is really simple: I'm losing log lines :P Most of the 
>> time everything goes well but when the logs are peaking high 1-5% 
>> logs are getting lost.
>>
>> Last night the stats of the server and a client said 0 drops but when 
>> I counted the lines I found lost lines. The server has 24g ram & 8 
>> cores and I can rule out a network problem for sure.
>>
>>
>>
>> So now to my questions, has anyone else an idea where I can tweak my 
>> cfg or where I have to look to find more clues? Is tcp the only way 
>> to get around it?
>>
>> I've attached my syslog server cfg. The so_rcvbuf buffer is the same 
>> size as the os net.core.rmem settings. And as described in the 
>> various balabit blog posts I played around with log_fetch_limit and flush_lines already.
>>
>>
>>
>> syslog-ng.conf:
>>
>> @version: 3.3
>>
>>
>>
>> options {
>>
>>     threaded(yes);
>>
>>     owner("root");
>>
>>     group("root");
>>
>>     perm(0660);
>>
>>
>>
>>     dir_owner("root");
>>
>>     dir_group("root");
>>
>>     dir_perm(0770);
>>
>>     create_dirs(yes);
>>
>>
>>
>>     stats_freq(600);
>>
>>     stats_level(2);
>>
>>     chain_hostnames(yes);
>>
>>     normalize_hostnames(yes);
>>
>>     check_hostname(yes);
>>
>>
>>
>>     dns_cache(yes);
>>
>>     dns_cache_size(16384);
>>
>>     dns_cache_expire(3600);
>>
>>     dns_cache_expire_failed(60);
>>
>>
>>
>>     log_msg_size(16384);
>>
>>     log_fifo_size(100000);
>>
>>
>>
>>
>>
>>     use_fqdn(yes);
>>
>> #disabled 4 debugging
>>
>> #    flush_lines(200);
>>
>> };
>>
>>
>>
>> source s_src {
>>
>>         unix-dgram("/dev/log");
>>
>>         internal();
>>
>>         file("/proc/kmsg" program_override("kernel"));
>>
>> };
>>
>>
>>
>> source s_net {
>>
>> udp(
>>
>>         log_fetch_limit(400)
>>
>>         so_rcvbuf(51200000)
>>
>>         keep_hostname(yes)
>>
>>         keep_timestamp(no)
>>
>>         ip("10.8.4.10")
>>
>>         port(514)
>>
>> );
>>
>> tcp(
>>
>>         so_rcvbuf(51200000)
>>
>>         so_keepalive(yes)
>>
>>         keep_hostname(no)
>>
>>         keep_timestamp(no)
>>
>>         ip("10.8.4.10")
>>
>>         port(514)
>>
>>
>>
>> );
>>
>> syslog();
>>
>> };
>>
>>
>>
>> filter f_syslog {
>>
>>      not program(access.log) and
>>
>>      not program(error.log) and
>>
>>      not program(beetle.log) and
>>
>>      not program(edge.log);
>>
>>
>>
>> };
>>
>>
>>
>> filter f_apache {
>>
>>     program(access.log) or
>>
>>     program(error.log);
>>
>> };
>>
>>
>>
>> filter f_applogs {
>>
>>     program(beetle.log)
>>
>>     or program(edge.log);
>>
>> };
>>
>>
>>
>> template t_plain {
>>
>>     template("$MSG\n"); template_escape(no);
>>
>> };
>>
>>
>>
>> destination d_messages { file("/var/log/messages"); };
>>
>> destination d_remote {
>> file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); };
>>
>> destination d_apache {
>> file("/log/apache/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
>> template(t_plain)); };
>>
>> destination d_applogs {
>> file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"
>> template(t_plain)); };
>>
>>
>>
>> log {
>>
>>     source(s_src);
>>
>>     destination(d_messages);
>>
>> };
>>
>>
>>
>> log {
>>
>>     source(s_net);
>>
>>     filter(f_syslog);
>>
>>     destination(d_remote);
>>
>> };
>>
>>
>>
>> log {
>>
>>     source(s_net);
>>
>>     filter(f_apache);
>>
>>     destination(d_apache);
>>
>> };
>>
>>
>>
>> log {
>>
>>     source(s_net);
>>
>>     filter(f_applogs);
>>
>>     destination(d_applogs);
>>
>> };
>>
>>
>>
>>
>>
>> Thanks
>>
>> Daniel Neubacher
>>
>>
>>
>>
>> _____________________________________________________________________
>> _
>> ________ Member info:
>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________
> ________ Member info: 
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: 
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
> ______________________________________________________________________
> ________ Member info: 
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: 
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq