Syslog-ng client through a load balancer with SSL/TLS encryption
I am trying to setup Syslog-ng to relay messages from one syslog server to another with a load balancer in between. I am also using TLS encryption. The issue I’m having right now is that when the client intiates the connection, it seems to lock on to a particular back end syslog server and send all of its messages there instead of switching off to another one. On its own this isn’t a big problem except that if that system goes down, the client doesn’t seem to be aware. I also haven’t found a good way to force syslog-ng to close and re-establish its connections without fully shutting down the relay system. We currently have no persistence setup on the load balancer. Is there a way to tell the relay server to periodically reconnect? Maybe send a certain amount of messages or data before reconnecting so that the data is balanced across the backend syslog-ng servers? Also, is there a better way to have the relay system learn about the remote server going offline so it can immediately reset its connection?
IMHO The best way to have redundant logging it to log to multiple syslog servers from each source server. For devices that can only log to one device I would log to a dedicated log replicator that send a copy of the log event to the multiple syslog servers just as if the client could have sent to multiple syslog server on its own. Evan. On 08/11/2016 03:07 PM, Lupo, Joseph wrote:
I am trying to setup Syslog-ng to relay messages from one syslog server to another with a load balancer in between. I am also using TLS encryption. The issue I’m having right now is that when the client intiates the connection, it seems to lock on to a particular back end syslog server and send all of its messages there instead of switching off to another one. On its own this isn’t a big problem except that if that system goes down, the client doesn’t seem to be aware. I also haven’t found a good way to force syslog-ng to close and re-establish its connections without fully shutting down the relay system. We currently have no persistence setup on the load balancer.
Is there a way to tell the relay server to periodically reconnect? Maybe send a certain amount of messages or data before reconnecting so that the data is balanced across the backend syslog-ng servers? Also, is there a better way to have the relay system learn about the remote server going offline so it can immediately reset its connection?
Multiple syslog servers isn’t an option with a lot of these systems. We could possibly have the relay server relay to multiple servers on the backend, but we’re loading this data into Splunk and don’t want redundant data to be loaded in. Thanks, Joe Lupo T-Mobile USA Principal Engineer, System Design & Strategy (973) 440-8768 From: <syslog-ng-bounces@lists.balabit.hu> on behalf of Evan Rempel <erempel@uvic.ca> Reply-To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Date: Thursday, August 11, 2016 at 11:19 PM To: "syslog-ng@lists.balabit.hu" <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Syslog-ng client through a load balancer with SSL/TLS encryption IMHO The best way to have redundant logging it to log to multiple syslog servers from each source server. For devices that can only log to one device I would log to a dedicated log replicator that send a copy of the log event to the multiple syslog servers just as if the client could have sent to multiple syslog server on its own. Evan. On 08/11/2016 03:07 PM, Lupo, Joseph wrote: I am trying to setup Syslog-ng to relay messages from one syslog server to another with a load balancer in between. I am also using TLS encryption. The issue I’m having right now is that when the client intiates the connection, it seems to lock on to a particular back end syslog server and send all of its messages there instead of switching off to another one. On its own this isn’t a big problem except that if that system goes down, the client doesn’t seem to be aware. I also haven’t found a good way to force syslog-ng to close and re-establish its connections without fully shutting down the relay system. We currently have no persistence setup on the load balancer. Is there a way to tell the relay server to periodically reconnect? Maybe send a certain amount of messages or data before reconnecting so that the data is balanced across the backend syslog-ng servers? Also, is there a better way to have the relay system learn about the remote server going offline so it can immediately reset its connection?
On Fri, Aug 12, 2016 at 03:47:43PM +0000, Lupo, Joseph wrote:
Multiple syslog servers isn’t an option with a lot of these systems. We could possibly have the relay server relay to multiple servers on the backend, but we’re loading this data into Splunk and don’t want redundant data to be loaded in.
FWIW one solution we're considering if our Elasticsearch cluster can handle the load is to push the logs twice but with the same ID: * no redundant data * possibility to track how many times the same log has been pushed to ES using the key '_version' Not sure that's possible using splunk though
syslog-ng at least in its open source incarnation does not support load balancing. the premium edition team has a feature that does fail-over to a list of servers, but that is yet to be ported over to the OSE version, even though the code is available. Here's the general description on how that works: https://www.balabit.com/documents/syslog-ng-pe-latest-guides/en/syslog-ng-pe... This is how it can be configured in Premium Edition: https://www.balabit.com/documents/syslog-ng-pe-latest-guides/en/syslog-ng-pe... On the open source side, it'd be great to have these features integrated to the open source tree, and it is something that Balabit is working on, but it's still time until those can be fully merged. With all this being said, you either have the option to approach Balabit with this request and continue the commercial route, OR help us porting the code in question OR you can wait until we get there. I can help with any of these :) Cheers, Bazsi -- Bazsi On Fri, Aug 12, 2016 at 12:07 AM, Lupo, Joseph <Joseph.Lupo@t-mobile.com> wrote:
I am trying to setup Syslog-ng to relay messages from one syslog server to another with a load balancer in between. I am also using TLS encryption. The issue I’m having right now is that when the client intiates the connection, it seems to lock on to a particular back end syslog server and send all of its messages there instead of switching off to another one. On its own this isn’t a big problem except that if that system goes down, the client doesn’t seem to be aware. I also haven’t found a good way to force syslog-ng to close and re-establish its connections without fully shutting down the relay system. We currently have no persistence setup on the load balancer.
Is there a way to tell the relay server to periodically reconnect? Maybe send a certain amount of messages or data before reconnecting so that the data is balanced across the backend syslog-ng servers? Also, is there a better way to have the relay system learn about the remote server going offline so it can immediately reset its connection?
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Evan Rempel
-
Fabien Wernli
-
Lupo, Joseph
-
Scheidler, Balázs