[syslog-ng] periodic disconnect of TCP sessions

[Laszlo Szemere (lszemere)]

Hello Jon,

 There is currently no way to deliberately terminate an active TCP session. What I was thinking about, that there might be a better approach to the original problem, and terminating the TCP session is just a fragile workaround.

 I have a couple of questions about your setup, so I can get a better understanding of the problem:

 - Is all of the clients controlled by you? (Maybe they are syslog-ng instances in the first place?)
 - Is there a particular reason to have multiple syslog-ng instances? At a first glance (with the parallel sending) it looked like a HA(ish) setup, but the later round-robin solution ruled it out. Also it looks like the DNS server is not aware of the states of the servers.
 - What is the current load on the system? Have you experienced any throttling or bottle neck. (Why do I ask? If the reason behind using multiple syslog-ng instances, is to load balance the traffic for the Splunk servers, than it can be achieved in a different way.
 - How many concurrent connections do you expect at the peak load?

Best regards,
Laci

________________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Wilson, Jonathan <jonathan.wilson at vumc.org>
Sent: Wednesday, November 13, 2019 23:03
To: syslog-ng at lists.balabit.hu
Subject: [syslog-ng] periodic disconnect of TCP sessions

CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.

Hello all,

I am using a pair of syslog-ng OSE 3.22.1 servers that write logfiles which are then scanned by a Splunk Universal Forwarder. They receive messages over TCP, TCP with TLS, and UDP. We have always had devices and systems that send us syslog messages simply send to both syslog-ng servers; however, this resulted in double indexing of the log data in Splunk.

To deal with this we set up a DNS name that round-robins across the two syslog-ng servers’ IPs every 30 seconds. The devices and systems that send to us now send to that DNS name. That neatly prevents the double indexing. If the messages are coming in over stateless UDP, the messages are load balanced in that they all go to one server for 30 seconds, then the other. However, TCP sessions are much longer lived, and some senders send many messages every second – they will latch onto one of our syslog servers and stay connected to it all day.

What I am looking for is a way to limit the lifetime of a TCP connection into syslog-ng, either by time or by number of messages received; after the connection is dropped, the sender will reconnect to whichever server is indicated by the round-robin DNS name, and over time about half of the messages will go to each server.

Is there already a way to do this? Failing that, can you suggest a place to start in patching the source?

Thanks,
Jon