RE: syslog-ng 2.0.4 How can syslog-ng achieve this performance???
Geller, Sandor Balazs Scheidler Thanks 4 ur advise. Please pardon me for my short information. The scenario is this, ----- There is a network device which sends logs to syslog server over network at a pace of approx 4000 logs/sec. Syslog server has its own mission to handle them. In case, syslog server must write logs to local disk, the server in fact drops some logs in the local file. Consequently, I turned to think of tuning some tweak in syslog-ng parameter or kernel parameter. I have tuned kernel parameter by setting "udp_recv_hiwat" to its maximum value, and udp_max_buf to the value of 300 times of the default value. And, I have come up with this idea. 1. Increase "sync" parameter to buffer some logs and write logs , not at the pace of every second. I tried to increase sync as well as log_fifo_size. First, I set sync as 3000 , log_fifo_size as 10000. However, it was not liked, with the message " The value of flush_lines must be less than fifo_size; fifo_size='1000',flush_lines='3000' ". Syslog-ng is configured as follows; sync (3000); time_reopen (10); time_sleep(0); log_fifo_size (10000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); source s_test { udp(ip(0.0.0.0) port(514)); }; destination d_local4 { file("/var/log/local4"); }; filter f_local4_al { facility(local4) and level(info) and match("xxxxxx") and filter(test); }; filter test { match("10600[1267]") or match("10601[0-8]") or match ("10602[0124567]") or match("106100") or match("20900[345]") or match("500004"); }; log { source(s_test); filter(f_local4_al); destination(d_local4); }; Any advice about how sync works , and how log_fifo_size works will greatly help. Is there any other way than editing the logwriter.c file and re-compile it? Is there difference between setting sync and log_fifo_size in global option section and individual destination section in terms of its effect? 2. "fsync" parameter would be thouhgt as second chance to overcome this problem. However, no userful information cound not be found on web. I set fsync in destination section, but it was rejected when reloading the process. Any advice about how fsync works will greatly help. In addition, how can I get STATS information of syslog-ng? I have added "stats_freq (60);" in global option section, but I could not get any information in /var/adm/messages. Where does syslog-ng output the stats information? Thanks! Regards George ------------------------------
Message: 3 Date: Wed, 20 Jun 2007 11:12:40 +0900 From: "Root Administrator" <root.regist@gmail.com> Subject: [syslog-ng] syslog-ng 2.0.4 How can syslog-ng achieve this performance??? To: syslog-ng@lists.balabit.hu Message-ID: <8ee7d730706191912l357ecabfgb137b7297c1bfbf0@mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1"
Hi All,
NEED HELP!!!!!
[Environment] SunOS 5.9 Generic_122300-07 sun4u sparc SUNW syslog-ng 2.0.4 disk : single disk (no RAID) syslog-ng conf (global option part) sync (3000); time_reopen (10); time_sleep(0); log_fifo_size (10000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes);
[NEED] I want syslog-ng to write logs to local disk at pace of about 4000 lines per second without any lines losing. However, lines were in fact lost in the local file. I am trying to know the syslog-ng max performable point.
[Consideration] "log_fifo_size" in global option is set as 10000. I tried to set sync() parameter, for instance 3000, in global option section. This did not succeed with this messages when reloading the process, " The value of flush_lines must be less than fifo_size; fifo_size='1000', flush_lines='3000' ". The administration guide says "sync" is alias for "flush_lines".
In addition, resource usage at 4000lines/s load was as below,
result of vmstat kthr memory page disk faults cpu r b w swap free re mf pi po fr de sr s3 sd sd -- in sy cs us sy id 0 0 0 3954792 3895288 155 8 353 0 0 0 12 0 2 0 0 669 4368 857 4 3 93 0 0 0 3955248 3926424 3 8 0 0 0 0 0 0 2 0 0 4112 27017 5889 21 15 64 0 0 0 3955248 3925664 0 0 0 0 0 0 0 0 3 0 0 4120 27251 5653 16 19 64 0 0 0 3955248 3924528 0 0 0 0 0 0 0 0 1 0 0 4129 27251 5914 15 18 67 0 0 0 3955248 3923400 0 0 0 0 0 0 0 0 1 0 0 4113 27236 6052 17 15 68 0 0 0 3955176 3922568 0 0 0 0 0 0 0 0 32 0 0 4156 26028 5405 19 13 68 0 0 0 3955176 3921808 0 0 0 0 0 0 0 0 1 0 0 4161 27316 5757 16 18 66 0 0 0 3955176 3921056 0 0 0 0 0 0 0 0 1 0 0 4120 27254 6136 14 18 68 0 0 0 3955176 3920296 0 0 0 0 0 0 0 0 1 0 0 4110 27244 5648 18 17 65 0 0 0 3955176 3919544 0 0 0 0 0 0 0 0 1 0 0 4115 27253 6042 17 19 64 0 0 0 3955176 3918784 0 0 0 0 0 0 0 0 1 0 0 4108 27238 6469 18 16 65 0 0 0 3955176 3918032 0 0 0 0 0 0 0 0 1 0 0 4107 27235 6106 16 18 66 0 0 0 3955176 3917272 0 0 0 0 0 0 0 0 1 0 0 4139 27264 5850 17 18 65 0 0 0 3955176 3916520 0 0 0 0 0 0 0 0 1 0 0 4107 27259 5867 19 14 67
result of iostat extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device 0.0 1.0 0.0 775.9 0.0 0.0 0.0 27.3 0 3 c1t0d0s3 1.0 2.0 8.0 744.1 0.0 0.0 0.0 11.4 0 3 c1t0d0s3 0.0 2.0 0.0 864.0 0.0 0.0 0.0 15.7 0 3 c1t0d0s3 0.0 4.0 0.0 856.0 0.0 0.1 0.0 17.3 0 3 c1t0d0s3 0.0 1.0 0.0 456.0 0.0 0.0 0.0 19.5 0 2 c1t0d0s3 0.0 1.0 0.0 856.1 0.0 0.0 0.0 28.1 0 3 c1t0d0s3 0.0 1.0 0.0 855.9 0.0 0.0 0.0 24.4 0 2 c1t0d0s3 0.0 1.0 0.0 856.1 0.0 0.0 0.0 27.3 0 3 c1t0d0s3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0s3 0.0 1.0 0.0 856.0 0.0 0.0 0.0 19.0 0 2 c1t0d0s3 0.0 1.0 0.0 856.0 0.0 0.0 0.0 26.9 0 3 c1t0d0s3 0.0 1.0 0.0 832.1 0.0 0.0 0.0 24.8 0 2 c1t0d0s3 0.0 1.0 0.0 735.9 0.0 0.0 0.0 26.4 0 3 c1t0d0s3 0.0 1.0 0.0 856.1 0.0 0.0 0.0 28.7 0 3 c1t0d0s3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0s3
[Question] Question 1. Does the "fifo_size" in above message mean "log_fifo_size" in global option ? Is the value "fifo_size='1000'" max value ? Is it possible to set "log_fifo_size" far more than 1000 ? Is it possible to set "sync" far more than 1000 ? If possible, then how do I do it ?
Question 2. To achieve the NEED, I am considering the parameters below, sync() log_fifo_size() . Are there any other parameters I MUST consider for syslog-ng configuration?
Regards
George
Geller, Sandor Balazs Scheidler
Thanks 4 ur advise.
Please pardon me for my short information.
The scenario is this, ----- There is a network device which sends logs to syslog server over network at a pace of approx 4000 logs/sec. Syslog server has its own mission to handle them. In case, syslog server must write logs to local disk, the server in fact drops some logs in the local file. Consequently, I turned to think of tuning some tweak in syslog-ng parameter or kernel parameter. I have tuned kernel parameter by setting "udp_recv_hiwat" to its maximum value, and udp_max_buf to the value of 300 times of the default value. And, I have come up with this idea.
1. Increase "sync" parameter to buffer some logs and write logs , not at the pace of every second. I tried to increase sync as well as log_fifo_size. First, I set sync as 3000 , log_fifo_size as 10000. However, it was not liked, with the message " The value of flush_lines must be less than fifo_size; fifo_size='1000',flush_lines='3000' ".
Syslog-ng is configured as follows; sync (3000); time_reopen (10); time_sleep(0); log_fifo_size (10000); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes);
source s_test { udp(ip(0.0.0.0) port(514)); }; destination d_local4 { file("/var/log/local4"); }; filter f_local4_al { facility(local4) and level(info) and match("xxxxxx") and filter(test); }; filter test { match("10600[1267]") or match("10601[0-8]") or match ("10602[0124567]") or match("106100") or match("20900[345]") or match("500004"); }; log { source(s_test); filter(f_local4_al); destination(d_local4); };
Your filters might cause CPU load... Is this heavy filtering really required? Usually not too many programs use local4 so you should try to examine whether this filtering is really need or not. Maybe you could turn the conditions to match to the logs which should be excluded. This kind of performance tuning could be done only when you have a representative set of data to experiment with.
Any advice about how sync works , and how log_fifo_size works will greatly help.
sync() is deprecated, use flush_lines() and flush_timeout() instead.
Is there any other way than editing the logwriter.c file and re-compile it?
Maybe a daily shapshot already contains the change.
Is there difference between setting sync and log_fifo_size in global option section and individual destination section in terms of its effect?
After applying the patch Bazsi sent there shouldn't be any difference.
2. "fsync" parameter would be thouhgt as second chance to overcome this problem. However, no userful information cound not be found on web. I set fsync in destination section, but it was rejected when reloading the process. Any advice about how fsync works will greatly help.
AFAIK fsync hasn't been implemented. Anyway it would cause performance problems, as fsync is used to make sure that the data ended up on solid storage. This means that the fsync call will block the caller until the kernel flushes VM/FS caches...
In addition, how can I get STATS information of syslog-ng? I have added "stats_freq (60);" in global option section, but I could not get any information in /var/adm/messages. Where does syslog-ng output the stats information?
Use the internal() source, it's missing from your configuration. Something like this: source s_internal { internal(); }; destination d_syslognglog { file("/var/log/syslog-ng.log"); }; log { source(s_internal); destination(d_syslognglog); }; Regards, Sandor -------------------------------------------------------- NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error.
participants (2)
-
Geller, Sandor (IT)
-
Root Administrator