[syslog-ng] Message loss (probably) within syslog-ng

[Paul Krizak]

Just as a point of reference I'm having the same problem with my 
configuration.  Virtually the same infrastructure, but about 3x the 
number of linux hosts dumping into syslog.  I hadn't taken the next step 
  of actually testing to see if packets were being dropped, etc., but 
the syslog-ng log reports lots of dropped messages despite using 
numerous filters to reduce the traffic going to logfiles and the database.

Paul Krizak                         5900 E. Ben White Blvd. MS 625
Advanced Micro Devices              Austin, TX  78741
Linux/Unix Systems Engineering      Phone: (512) 602-8775
Microprocessor Solutions Sector


Vincent Haverlant wrote:
> Le Mon Mar  6 08:07:51 2006, Mike a écrit:
> | 
> | >The default receive buffer size is set using the parameter
> | >udp_recv_hiwat which is currently set to its maximum value of 65536. To
> | >increase it the only way is the setsockopt syscall.
> | >
> | >Anyway I don't get that many udpInOverflows (a few everyday) but it does
> | >not explain the few thousand messages I miss in my logs files.
> | >
> | >Regards,
> | >Vincent.
> | 
> | 
> | hmmm. strange..on Linux it seems to use that max value without modifying applications. I could be wrong tho.
> | 
> 
> I think it is the same issue on linux: (extract from man 7 socket)
>    SO_RCVBUF
>        Sets  or gets the maximum socket receive buffer in bytes.
>        The default value is set by the rmem_default sysctl
>        and the maximum allowed value is set by the rmem_max
>        sysctl.
> 
> The buffer size is controled by two parameters: rmem_default and
> rmem_max. The difference with Solaris is that these two parameters are
> global to all ip protocols except set otherwise for tcp with tcp_rmem.
> 
> | options {
> | .
> | .
> | .
> |  use_dns (yes);
> |  dns_cache (yes);
> |  dns_cache_size(3000);
> |  use_fqdn (no); # utilisation du nom court de la machine
> | .
> | .
> | .
> | .
> | };
> | 
> | hmm..I am kinda wondering about the DNS usage now tho. I have never used this feature before, but from what the docs say, syslog-ng will block on DNS 
> | queries...can you tell if any of DNS queries are failing?
> 
> I saw that too, unfortunately even after turning it off, I still miss
> between 5 to 15% messages in my test, which I will admit is a burst test
> but bursts can happen when you have 2500 hosts.
> 
> | maybe you could add in some options here:
> |  dns_cache_expire(n)
> |            Number of seconds while a successful lookup is cached.
> | 
> | 
> |       dns_cache_expire_failed(n)
> |            Number of seconds while a failed lookup is cached.
> | 
> | but if syslog-ng blocks on DNS queries, I would imagine that you would see your udpInOverflows value increase....
> | 
> | anyone know the default value of dns_cache_expire() off the top of your heads?
> 
> dns_cache_expire -> 3600
> dns_cache_expire_failed -> 60
> 
> I'm quite at loss as to what to do now...
> Vincent.