Load balancing in syslog-ng(verison 3.0)
Hi, In my syslog-ng server configuration file I want to divide the messages in 2 different pipes to share the load. Please let me know how to do this ? Thanks Vaibhav
"Jain, Vaibhav (GE Healthcare)" <Vaibhav.Jain@ge.com> writes:
In my syslog-ng server configuration file I want to divide the messages in 2 different pipes to share the load. Please let me know
The simplest approach would be to filter the messages, based on a condition that would evenly distribute them (what such a condition is, depends on your logs, and it may not even exist), and send them towards different destinations. For example, assuming you have two client hosts, src1 and src2, and two destinations: dst1, and dst2, your config could look like this: source s_network { tcp(); }; destination d_dst1 { tcp("dst1.local"); }; destination d_dst2 { tcp("dst2.local"); }; filter f_host1 { host("src1"); }; log { source(s_network); filter(f_host1); destination(d_dst1); flags(final); }; log { source(s_network); destination(d_dst2); }; -- |8]
Hi, Thanks for your quick response. In my case logs are coming from one source machine. Let me know how to put the filter condition for sharing the load. Regards, Vaibhav -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Thursday, September 22, 2011 5:29 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Load balancing in syslog-ng(verison 3.0) "Jain, Vaibhav (GE Healthcare)" <Vaibhav.Jain@ge.com> writes:
In my syslog-ng server configuration file I want to divide the messages in 2 different pipes to share the load. Please let me know
The simplest approach would be to filter the messages, based on a condition that would evenly distribute them (what such a condition is, depends on your logs, and it may not even exist), and send them towards different destinations. For example, assuming you have two client hosts, src1 and src2, and two destinations: dst1, and dst2, your config could look like this: source s_network { tcp(); }; destination d_dst1 { tcp("dst1.local"); }; destination d_dst2 { tcp("dst2.local"); }; filter f_host1 { host("src1"); }; log { source(s_network); filter(f_host1); destination(d_dst1); flags(final); }; log { source(s_network); destination(d_dst2); }; -- |8] ________________________________________________________________________ ______ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
"Jain, Vaibhav (GE Healthcare)" <Vaibhav.Jain@ge.com> writes:
Thanks for your quick response. In my case logs are coming from one source machine. Let me know how to put the filter condition for sharing the load.
Like I said, you'll need to find a filter with which to split the logs. Preferably reasonably evenly. Whether such a filter exists, or what it is, depends on your logs. -- |8]
http://www.cisco.com/en/US/products/ps8361/index.html Buy an ACE and be done with it! :-D On 09/22/2011 09:00 AM, Gergely Nagy wrote:
"Jain, Vaibhav (GE Healthcare)"<Vaibhav.Jain@ge.com> writes:
Thanks for your quick response. In my case logs are coming from one source machine. Let me know how to put the filter condition for sharing the load.
Like I said, you'll need to find a filter with which to split the logs. Preferably reasonably evenly.
Whether such a filter exists, or what it is, depends on your logs.
Brandon Phelps <bphelps@gls.com> writes:
http://www.cisco.com/en/US/products/ps8361/index.html
Buy an ACE and be done with it! :-D
Well, I could suggest cheaper alternatives. Like logging to mongodb, and sharding the database: it's free, reasonably easy to set up, and you don't need extra hardware apart from the database servers. ;] (You do need a more recent syslog-ng, though. But I believe that's a very small price to pay ;) -- |8]
Check your logs and see which characteristics can be used to balance them. If they are coming from a single host, try to filter by program name, or facility. HTH Robert On 09/22/2011 02:33 PM, Jain, Vaibhav (GE Healthcare) wrote:
Hi,
Thanks for your quick response. In my case logs are coming from one source machine. Let me know how to put the filter condition for sharing the load.
Regards, Vaibhav
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Thursday, September 22, 2011 5:29 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Load balancing in syslog-ng(verison 3.0)
"Jain, Vaibhav (GE Healthcare)"<Vaibhav.Jain@ge.com> writes:
In my syslog-ng server configuration file I want to divide the messages in 2 different pipes to share the load. Please let me know
The simplest approach would be to filter the messages, based on a condition that would evenly distribute them (what such a condition is, depends on your logs, and it may not even exist), and send them towards different destinations.
For example, assuming you have two client hosts, src1 and src2, and two destinations: dst1, and dst2, your config could look like this:
source s_network { tcp(); }; destination d_dst1 { tcp("dst1.local"); }; destination d_dst2 { tcp("dst2.local"); };
filter f_host1 { host("src1"); };
log { source(s_network); filter(f_host1); destination(d_dst1); flags(final); }; log { source(s_network); destination(d_dst2); };
You could try a match() with $SEQNUM which is an ever-increasing counter. A regex that looked for even $SEQNUM's would perfectly load balance. On Thu, Sep 22, 2011 at 9:17 AM, Fekete Robert <frobert@balabit.hu> wrote:
Check your logs and see which characteristics can be used to balance them. If they are coming from a single host, try to filter by program name, or facility.
HTH
Robert
On 09/22/2011 02:33 PM, Jain, Vaibhav (GE Healthcare) wrote:
Hi,
Thanks for your quick response. In my case logs are coming from one source machine. Let me know how to put the filter condition for sharing the load.
Regards, Vaibhav
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Thursday, September 22, 2011 5:29 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Load balancing in syslog-ng(verison 3.0)
"Jain, Vaibhav (GE Healthcare)"<Vaibhav.Jain@ge.com> writes:
In my syslog-ng server configuration file I want to divide the messages in 2 different pipes to share the load. Please let me know
The simplest approach would be to filter the messages, based on a condition that would evenly distribute them (what such a condition is, depends on your logs, and it may not even exist), and send them towards different destinations.
For example, assuming you have two client hosts, src1 and src2, and two destinations: dst1, and dst2, your config could look like this:
source s_network { tcp(); }; destination d_dst1 { tcp("dst1.local"); }; destination d_dst2 { tcp("dst2.local"); };
filter f_host1 { host("src1"); };
log { source(s_network); filter(f_host1); destination(d_dst1); flags(final); }; log { source(s_network); destination(d_dst2); };
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
But not everything sends a sequence number (or does syslog-ng have an internal one?) ______________________________________________________________ Clayton Dukes ______________________________________________________________ On Fri, Sep 23, 2011 at 1:37 PM, Martin Holste <mcholste@gmail.com> wrote:
You could try a match() with $SEQNUM which is an ever-increasing counter. A regex that looked for even $SEQNUM's would perfectly load balance.
On Thu, Sep 22, 2011 at 9:17 AM, Fekete Robert <frobert@balabit.hu> wrote:
Check your logs and see which characteristics can be used to balance them. If they are coming from a single host, try to filter by program name, or facility.
HTH
Robert
On 09/22/2011 02:33 PM, Jain, Vaibhav (GE Healthcare) wrote:
Hi,
Thanks for your quick response. In my case logs are coming from one source machine. Let me know how to put the filter condition for sharing the load.
Regards, Vaibhav
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Gergely Nagy Sent: Thursday, September 22, 2011 5:29 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Load balancing in syslog-ng(verison 3.0)
"Jain, Vaibhav (GE Healthcare)"<Vaibhav.Jain@ge.com> writes:
In my syslog-ng server configuration file I want to divide the messages in 2 different pipes to share the load. Please let me know
The simplest approach would be to filter the messages, based on a condition that would evenly distribute them (what such a condition is, depends on your logs, and it may not even exist), and send them towards different destinations.
For example, assuming you have two client hosts, src1 and src2, and two destinations: dst1, and dst2, your config could look like this:
source s_network { tcp(); }; destination d_dst1 { tcp("dst1.local"); }; destination d_dst2 { tcp("dst2.local"); };
filter f_host1 { host("src1"); };
log { source(s_network); filter(f_host1); destination(d_dst1); flags(final); }; log { source(s_network); destination(d_dst2); };
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
On Fri, 2011-09-23 at 13:49 -0400, Clayton Dukes wrote:
But not everything sends a sequence number (or does syslog-ng have an internal one?)
Yup, SEQNUM is related to the structured data [meta sequenceId] (defined by RFC5424. It is defined to be a counter on a hop-by-hop basis, so it is only defined at the destination side (and is different for different tcp destinations even for the same message). I have $RCPTID in mind, which would be unique for each received message, but that doesn't exist yet. If using 3.3, it'd be very simple to create a filter plugin that distributes messages evenly. -- Bazsi
participants (7)
-
Balazs Scheidler
-
Brandon Phelps
-
Clayton Dukes
-
Fekete Robert
-
Gergely Nagy
-
Jain, Vaibhav (GE Healthcare)
-
Martin Holste