[syslog-ng] Single destination, multiple tcp connections?

[Balazs Scheidler]

On Fri, Mar 24, 2023, 18:05 Steve Bernacki <steve at copacetic.net> wrote:

> I am running syslog-ng 3.38.1.
>
> I have a scenario where I am receiving logs from hundreds devices over a
> number of different protocols (tcp, udp, tcp/TLS). I perform some
> filtering on these logs and then send them to one or more destinations
> for further processing. Due to the volume of logs being received, the
> destinations are becoming saturated and logs begin queuing up on my end.
> I have memory and disk buffering enabled, but the receiving end isn't
> able to pull logs off quickly enough since they are being funneled
> through a single TCP connection. I'd like to be able to establish some
> number of concurrent tcp connections to a single destination ip:port,
> and balance all of the incoming logs through those connections.
>
> I'm aware of techniques to load balance to destinations using multiple
> channels in a single destination and filtering traffic by the R_MSEC,
> but this technique quickly causes simple configs to balloon to hundreds
> of lines. I suppose this would work even if the destinations are all the
> same (I'd need to use unique persist-name() labels), but is there an
> easier / more straightforward way of accomplishing this?
>


Just one idea to throw around, you can generate syslog-ng config using
confgen (using a Shell script in earlier versions or a python function
since 4.0). This would mean that your primary config would only contain a
single destination and behind that  you can have the entire load balancing
logic.

Here's a sample for a python based confgen from the 4.0 announcement.

https://github.com/syslog-ng/syslog-ng/releases/tag/syslog-ng-4.0.1

Bazsi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20230325/518f8528/attachment.htm>