[syslog-ng] Losing to much remote sent logs

[Balazs Scheidler]

On Fri, 2012-03-02 at 10:40 +0100, Daniel Neubacher wrote:
> Hello there,
> 
> I’ve started playing around with syslog-ng 3.3.4 ose a few days ago
> but I’m still experiencing some trouble. First of all we want to use
> syslog-ng to send all of our logs via udp to a central syslog server.
> This includes of course syslogs, apache logs and custom generated
> applogs. These logs are generated from 400 clients and produces a
> minimum of 300 mio. log lines a day. 
> 
> The problem is really simple: I’m losing log lines :P Most of the time
> everything goes well but when the logs are peaking high 1-5% logs are
> getting lost. 
> 
> Last night the stats of the server and a client said 0 drops but when
> I counted the lines I found lost lines. The server has 24g ram & 8
> cores and I can rule out a network problem for sure. 
> 
>  
> 
> So now to my questions, has anyone else an idea where I can tweak my
> cfg or where I have to look to find more clues? Is tcp the only way to
> get around it?
> 
> I’ve attached my syslog server cfg. The so_rcvbuf buffer is the same
> size as the os net.core.rmem settings. And as described in the various
> balabit blog posts I played around with log_fetch_limit and
> flush_lines already.

I've received reports, that syslog-ng's latency over UDP sources became
worse with the threaded architecture and threaded(yes) is specified.

One possible workaround until I get around to look into the issue a bit
deeper is to disable the global threaded() option and enable the
"threaded" flag on all non-udp sources and all destinations. This should
basically be the same, except that udp traffic will be processed in the
main thread, which effectively reduces the latency for the udp source.

I saw that you rolled out tcp() already, but this information could help
someone else trying to tackle the same issue in the future.


-- 
Bazsi