----- Original message -----
> Any suggestions on how I debug this?
>
> I have a tcp and a udp source (defined as separate sources) that share a
> common destination
>
> log { souce(s_udp); source(s_tcp); destination(d_file); };
>
> 90% of the log volume comes from the tcp sources.
> The udp log lines are logged 10 minutes "late". If I decrease the
> so_rcvbuf then I can get this down to 5-10 seconds "late." I can issue
> on a source udp server
>
> logger -t test "this is a test"
>
> and wait on the server. It shows up, but minutes later.
>
> OK, playing a little more, if I add the log_iw_size option on the
> udp source, the symptoms go away. Odd since I do NOT have any flow
> control in any destination.
>
> Can anyone explain why that option would make a difference?

hmmm,, with 3.3 flow control behaviour was changed a bit because of the multithreaded nature of syslog-ng. file destinations enable a mode I called soft-flow-control.

In 3.2 and earlier, syslog-ng always prioritized output over input, effectively starving all inputs as long as the outputs were busy flushing data.

The same behaviour was unfeasible with the multithreaded architecture, so I employed the flow control logic to control the input flow (that was its role after all). However people usually don't have flags(flow-control) in their configuration, removing the architectural guarantee to prioritize output over inputs would cause message loss in cases there were none before.

Therefore file destinations automatically enable soft-flow-control, which means that as long as there's an active file opened, flow control is in operation. If syslog-ng fails to open a file, or it gets suspended because of a write error (e.g. disk full) flow control is lifted again.

This way, log-iw-size() does have an effect after all, as long as you have a file destination in the log path.

The log-iw-size and log-fifo-size defaults were also increased in a number of cases, but maybe not for udp.

What may be biting you is a small window size and a busy file destination, which takes minutes to flush its buffered data (that must be busy indeed) Try increasing flush-lines for the file destination, it should increase the output bandwidth a lot.

Probably the so-rcvbuf() only decreased the amount of lost udp frames, the one delayed was probably lost before.

hope this helps.