----- Original message -----
Any suggestions on how I debug this?
I have a tcp and a udp source (defined as separate sources) that share a common destination
log { souce(s_udp); source(s_tcp); destination(d_file); };
90% of the log volume comes from the tcp sources. The udp log lines are logged 10 minutes "late". If I decrease the so_rcvbuf then I can get this down to 5-10 seconds "late." I can issue on a source udp server
logger -t test "this is a test"
and wait on the server. It shows up, but minutes later.
OK, playing a little more, if I add the log_iw_size option on the udp source, the symptoms go away. Odd since I do NOT have any flow control in any destination.
Can anyone explain why that option would make a difference?
hmmm,, with 3.3 flow control behaviour was changed a bit because of the multithreaded nature of syslog-ng. file destinations enable a mode I called soft-flow-control. In 3.2 and earlier, syslog-ng always prioritized output over input, effectively starving all inputs as long as the outputs were busy flushing data. The same behaviour was unfeasible with the multithreaded architecture, so I employed the flow control logic to control the input flow (that was its role after all). However people usually don't have flags(flow-control) in their configuration, removing the architectural guarantee to prioritize output over inputs would cause message loss in cases there were none before. Therefore file destinations automatically enable soft-flow-control, which means that as long as there's an active file opened, flow control is in operation. If syslog-ng fails to open a file, or it gets suspended because of a write error (e.g. disk full) flow control is lifted again. This way, log-iw-size() does have an effect after all, as long as you have a file destination in the log path. The log-iw-size and log-fifo-size defaults were also increased in a number of cases, but maybe not for udp. What may be biting you is a small window size and a busy file destination, which takes minutes to flush its buffered data (that must be busy indeed) Try increasing flush-lines for the file destination, it should increase the output bandwidth a lot. Probably the so-rcvbuf() only decreased the amount of lost udp frames, the one delayed was probably lost before. hope this helps.