syslog-ng 2 and bandwidth limiting
Hi, Is there a way to limit the bandwidth of an ng server? If so, does it work with UDP or only TCP? I have seen the log_fifo_size() option, but don't easily see how this would limit the amount of bandwidth used (seems like it is designed to provide a larger output buffer for slow links, but does not actually limit the bandwidth used by NG) I am also looking at the flow-control flag, but again it is not clear how to use it to limit the bandwidth. Ideally I would like something that would allow NG to be limited to a specific amount of bandwidth (packets/sec or kb/s, etc.) for each log path. This way if a service started generating an excessive amount of log data (e.g. if a daemon were put into debug mode) it would help prevent the log server from choking the network trying to send all the packets (it would be especially nice if you could configure how large an output buffer were available, so if you wanted to deal with *bursty* services, you could limit them to a small amount of bandwidth, but provide them with a larger output buffer so you would be less likely to drop messages. Thoughts? Thanks! Jim Hendrick
On Thu, 2007-04-05 at 12:07 -0400, jrhendri@maine.rr.com wrote:
Hi, Is there a way to limit the bandwidth of an ng server?
If so, does it work with UDP or only TCP?
I have seen the log_fifo_size() option, but don't easily see how this would limit the amount of bandwidth used (seems like it is designed to provide a larger output buffer for slow links, but does not actually limit the bandwidth used by NG)
I am also looking at the flow-control flag, but again it is not clear how to use it to limit the bandwidth.
Ideally I would like something that would allow NG to be limited to a specific amount of bandwidth (packets/sec or kb/s, etc.) for each log path. This way if a service started generating an excessive amount of log data (e.g. if a daemon were put into debug mode) it would help prevent the log server from choking the network trying to send all the packets (it would be especially nice if you could configure how large an output buffer were available, so if you wanted to deal with *bursty* services, you could limit them to a small amount of bandwidth, but provide them with a larger output buffer so you would be less likely to drop messages.
syslog-ng has no such feature currently. albeit it would not be difficult to implement, at least on a per-destination basis. On a per-application basis, this would be a bit more complex. -- Bazsi
Thanks for the response. I am not familiar enough with the overall architecture to know where to look to implement something like this (nor have I written any significant code in several years). Does the message flow create a separate network connection for each 'destination' or would it be at the 'log' level? Does it maintain a separate buffer (queue) at the destination level or log level? I was thinking that one feature that would probably make this more useful would be an internal (or local) alert message related to bandwidth. Something like if the bandwidth exceeds the cap for a given destination send an alert message (once per time interval). That way you would know (downstream) that log messages were (possibly) being lost, or at least that a bandwidth condition had been reached. Thoughts? And thanks again for the reply! Jim
-----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Friday, April 06, 2007 5:19 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng 2 and bandwidth limiting
On Thu, 2007-04-05 at 12:07 -0400, jrhendri@maine.rr.com wrote:
Hi, Is there a way to limit the bandwidth of an ng server? Ideally I would like something that would allow NG to be limited to a specific amount of bandwidth (packets/sec or kb/s, etc.) for each log path.
syslog-ng has no such feature currently. albeit it would not be difficult to implement, at least on a per-destination basis. On a per-application basis, this would be a bit more complex.
-- Bazsi
_______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/sysl> og-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
On Fri, 2007-04-06 at 06:11 -0400, Jim Hendrick wrote:
Thanks for the response. I am not familiar enough with the overall architecture to know where to look to implement something like this (nor have I written any significant code in several years).
Does the message flow create a separate network connection for each 'destination' or would it be at the 'log' level?
Each and every destination network connection has a separate queue. No other queueing is done inside syslog-ng. (e.g. tcp driver has one, which is kept when the connection is broken)
Does it maintain a separate buffer (queue) at the destination level or log level?
per-destination, multiple log statements can refer to the same destination.
I was thinking that one feature that would probably make this more useful would be an internal (or local) alert message related to bandwidth. Something like if the bandwidth exceeds the cap for a given destination send an alert message (once per time interval).
That way you would know (downstream) that log messages were (possibly) being lost, or at least that a bandwidth condition had been reached.
There is currently a "log statistics" message that reports various counters, among others it has a 'dropped' counter for every destination. -- Bazsi
Hi, Can some one point me to syslog-ng 2 installation document. In the documentation section I only see syslog-ng 1.6_10 installation guide. I am looking for pre-requisites of this installation like Support on solaris 10, any patches need to be installed, etc.., Thanks in advance. Thanks, Ravi Kumar P.
Hi, Can someone give reference to syslog-ng2.0.3 installation guide and pre-requisites. Thanks, Ravi Kumar P. -----Original Message----- From: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Ravi Papisetti -X (rpapiset - HCL at Cisco) Sent: Tuesday, April 24, 2007 10:35 AM To: Syslog-ng users' and developers' mailing list Subject: [syslog-ng] Installation guide for syslog-ng 2 Hi, Can some one point me to syslog-ng 2 installation document. In the documentation section I only see syslog-ng 1.6_10 installation guide. I am looking for pre-requisites of this installation like Support on solaris 10, any patches need to be installed, etc.., Thanks in advance. Thanks, Ravi Kumar P. _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Hi, I am trying to configure syslog-ng 2.0.3 on solaris 10 and below is the error checking for EVTLOG... Package eventlog was not found in the pkg-config search path. Perhaps you should add the directory containing `eventlog.pc' to the PKG_CONFIG_PATH environment variable No package 'eventlog' found configure: error: Cannot find eventlog version >= 0.2: is pkg-config in path? I have installed libnet-1.20 and libol-0.3.18, then ran ./configure from /syslog-ng-2.0.3. Am I missing something? Thanks, Ravi Kumar P.
Hi, I see below error when I run make && make install make all-recursive make[1]: Entering directory `/usr/local/syslog-ng/syslog-ng-2.0.3' Making all in src make[2]: Entering directory `/usr/local/syslog-ng/syslog-ng-2.0.3/src' if gcc -DHAVE_CONFIG_H -I. -I. -I.. -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include -I/usr/local/include/eventlog -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -g -O2 -Wall -g -MT cfg-grammar.o -MD -MP -MF ".deps/cfg-grammar.Tpo" \ -c -o cfg-grammar.o `test -f 'cfg-grammar.c' || echo './'`cfg-grammar.c; \ then mv -f ".deps/cfg-grammar.Tpo" ".deps/cfg-grammar.Po"; \ else rm -f ".deps/cfg-grammar.Tpo"; exit 1; \ fi In file included from /home/bazsi/zwa/work/syslog-ng-2.0/syslog-ng/src/cfg-grammar.y:15: afinet.h:30:20: libnet.h: No such file or directory In file included from /home/bazsi/zwa/work/syslog-ng-2.0/syslog-ng/src/cfg-grammar.y:15: afinet.h:61: error: syntax error before "libnet_t" afinet.h:61: warning: no semicolon at end of struct or union afinet.h:63: warning: type defaults to `int' in declaration of `AFInetDestDriver' afinet.h:63: warning: data definition has no type or storage class /home/bazsi/zwa/work/syslog-ng-2.0/syslog-ng/src/cfg-grammar.y: In function `yyparse': /home/bazsi/zwa/work/syslog-ng-2.0/syslog-ng/src/cfg-grammar.y:592: error: syntax error before ')' token /home/bazsi/zwa/work/syslog-ng-2.0/syslog-ng/src/cfg-grammar.y:626: error: syntax error before ')' token make[2]: *** [cfg-grammar.o] Error 1 make[2]: Leaving directory `/usr/local/syslog-ng/syslog-ng-2.0.3/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/usr/local/syslog-ng/syslog-ng-2.0.3' make: *** [all] Error 2 Some one could tell me the reason. Thanks, Ravi Kumar P.
jrhendri@maine.rr.com wrote:
Hi, Is there a way to limit the bandwidth of an ng server?
If so, does it work with UDP or only TCP?
Why not put this problem where it really belongs, and state that it's a network problem - and needs a network-based solution? i.e. why can't your "network people" implement TOS/packet shaping to limit the max bandwidth TCP port 514 can use? ...and yes, I know that *for individual cases* it's easier to try to solve this at an application layer, but it doesn't change the fact that this is a network problem. Not all network applications have bandwidth limiting features, and yet *all network applications* have the potential to swallow all bandwidth. So the real solution is at the network layer - not the application layer. Hmmm, actually - there is a hybrid solution: implement TOS on the servers IP stack. i.e. tell the server that's running syslog-ng to limit how much bandwidth it's using. Great thing is you can do that for any network protocol - not just syslog-ng. -- Cheers Jason Haar Information Security Manager, Trimble Navigation Ltd. Phone: +64 3 9635 377 Fax: +64 3 9635 417 PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
participants (5)
-
Balazs Scheidler
-
Jason Haar
-
Jim Hendrick
-
jrhendri＠maine.rr.com
-
Ravi Papisetti -X (rpapiset - HCL＠Cisco)