I am running syslog-ng 3.38.1.
I have a scenario where I am receiving logs from hundreds devices over a
number of different protocols (tcp, udp, tcp/TLS). I perform some
filtering on these logs and then send them to one or more destinations
for further processing. Due to the volume of logs being received, the
destinations are becoming saturated and logs begin queuing up on my end.
I have memory and disk buffering enabled, but the receiving end isn't
able to pull logs off quickly enough since they are being funneled
through a single TCP connection. I'd like to be able to establish some
number of concurrent tcp connections to a single destination ip:port,
and balance all of the incoming logs through those connections.
I'm aware of techniques to load balance to destinations using multiple
channels in a single destination and filtering traffic by the R_MSEC,
but this technique quickly causes simple configs to balloon to hundreds
of lines. I suppose this would work even if the destinations are all the
same (I'd need to use unique persist-name() labels), but is there an
easier / more straightforward way of accomplishing this?
Just one idea to throw around, you can generate syslog-ng config using confgen (using a Shell script in earlier versions or a python function since 4.0). This would mean that your primary config would only contain a single destination and behind that you can have the entire load balancing logic.
Here's a sample for a python based confgen from the 4.0 announcement.
Bazsi