<div><div>Geller, Sandor<br>Balazs Scheidler<br><br>Thanks 4 ur advise.<br> </div>Please pardon me for my short information.<br><br>The scenario is this,<br>-----<br>There is a network device which sends logs to syslog server over network at a pace of approx 4000 logs/sec.
<br>Syslog server has its own mission to handle them.<br>In case, syslog server must write logs to local disk, the server in fact drops some logs in the local file.<br>Consequently, I turned to think of tuning some tweak in syslog-ng parameter or kernel parameter.
<br>I have tuned kernel parameter by setting "udp_recv_hiwat" to its maximum value, and udp_max_buf to the value of 300 times of the default value.<br>And, I have come up with this idea.<br><br> 1. Increase "sync" parameter to buffer some logs and write logs , not at the pace of every second.
<br> I tried to increase sync as well as log_fifo_size.<br> First, I set sync as 3000 , log_fifo_size as 10000.<br> However, it was not liked, with the message <br> " The value of flush_lines must be less than fifo_size; fifo_size='1000',flush_lines='3000' ".
<br><br> Syslog-ng is configured as follows;<br> sync (3000);<br> time_reopen (10);<br> time_sleep(0);<br> log_fifo_size (10000);<br> long_hostnames (off);<br> use_dns (no);
<br> use_fqdn (no);<br> create_dirs (no);<br> keep_hostname (yes);<br><br> source s_test { udp(ip(<a href="http://0.0.0.0">0.0.0.0</a>) port(514)); };<br> destination d_local4 { file("/var/log/local4"); };
<br> filter f_local4_al { facility(local4) and level(info) and match("xxxxxx") and filter(test); };<br> filter test { match("10600[1267]") or match("10601[0-8]") or match ("10602[0124567]") or match("106100") or
<br> match("20900[345]") or match("500004"); };<br> log { source(s_test); filter(f_local4_al); destination(d_local4); };<br><br> Any advice about how sync works , and how log_fifo_size works will greatly help.
<br>
Is there any other way than editing the logwriter.c file and re-compile it?<br> Is there difference between setting sync and log_fifo_size in global option section and individual destination section<br> in terms of its effect?
<br><br> 2. "fsync" parameter would be thouhgt as second chance to overcome this problem.<br> However, no userful information cound not be found on web.<br> I set fsync in destination section, but it was rejected when reloading the process.
<br> Any advice about how fsync works will greatly help.<br><br>In addition, how can I get STATS information of syslog-ng?<br>I have added "stats_freq (60);" in global option section, but I could not get any information in /var/adm/messages.
<br>Where does syslog-ng output the stats information?<br><br>Thanks!<br><br>Regards<br><br>George<br><br><br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
------------------------------<br><br>Message: 3<br>Date: Wed, 20 Jun 2007 11:12:40 +0900<br>From: "Root Administrator" <<a href="mailto:root.regist@gmail.com">root.regist@gmail.com</a>><br>Subject: [syslog-ng] syslog-ng
2.0.4 How can syslog-ng achieve this<br> performance???<br>To: <a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><br>Message-ID:<br> <<a href="mailto:8ee7d730706191912l357ecabfgb137b7297c1bfbf0@mail.gmail.com">
8ee7d730706191912l357ecabfgb137b7297c1bfbf0@mail.gmail.com</a>><br>Content-Type: text/plain; charset="iso-8859-1"<br><br>Hi All,<br><br>NEED HELP!!!!!<br><br>[Environment]<br>SunOS 5.9 Generic_122300-07 sun4u sparc SUNW
<br>syslog-ng 2.0.4<br>disk : single disk (no RAID)<br>syslog-ng conf (global option part)<br> sync (3000);<br> time_reopen (10);<br> time_sleep(0);<br> log_fifo_size (10000);<br> long_hostnames (off);
<br> use_dns (no);<br> use_fqdn (no);<br> create_dirs (no);<br> keep_hostname (yes);<br><br>[NEED]<br>I want syslog-ng to write logs to local disk at pace of about 4000<br>lines per second without any lines losing.
<br>However, lines were in fact lost in the local file.<br>I am trying to know the syslog-ng max performable point.<br><br>[Consideration]<br>"log_fifo_size" in global option is set as 10000.<br>I tried to set sync() parameter, for instance 3000, in global option
<br>section.<br>This did not succeed with this messages when reloading the process,<br>" The value of flush_lines must be less than fifo_size; fifo_size='1000',<br>flush_lines='3000' ".<br>The administration guide says "sync" is alias for "flush_lines".
<br><br>In addition, resource usage at 4000lines/s load was as below,<br><br>result of vmstat<br> kthr memory page disk faults cpu<br> r b w swap free re mf pi po fr de sr s3 sd sd -- in sy cs us sy
<br>id<br> 0 0 0 3954792 3895288 155 8 353 0 0 0 12 0 2 0 0 669 4368 857 4 3<br>93<br> 0 0 0 3955248 3926424 3 8 0 0 0 0 0 0 2 0 0 4112 27017 5889 21 15<br>64<br> 0 0 0 3955248 3925664 0 0 0 0 0 0 0 0 3 0 0 4120 27251 5653 16 19
<br>64<br> 0 0 0 3955248 3924528 0 0 0 0 0 0 0 0 1 0 0 4129 27251 5914 15 18<br>67<br> 0 0 0 3955248 3923400 0 0 0 0 0 0 0 0 1 0 0 4113 27236 6052 17 15<br>68<br> 0 0 0 3955176 3922568 0 0 0 0 0 0 0 0 32 0 0 4156 26028 5405 19 13
<br>68<br> 0 0 0 3955176 3921808 0 0 0 0 0 0 0 0 1 0 0 4161 27316 5757 16 18<br>66<br> 0 0 0 3955176 3921056 0 0 0 0 0 0 0 0 1 0 0 4120 27254 6136 14 18<br>68<br> 0 0 0 3955176 3920296 0 0 0 0 0 0 0 0 1 0 0 4110 27244 5648 18 17
<br>65<br> 0 0 0 3955176 3919544 0 0 0 0 0 0 0 0 1 0 0 4115 27253 6042 17 19<br>64<br> 0 0 0 3955176 3918784 0 0 0 0 0 0 0 0 1 0 0 4108 27238 6469 18 16<br>65<br> 0 0 0 3955176 3918032 0 0 0 0 0 0 0 0 1 0 0 4107 27235 6106 16 18
<br>66<br> 0 0 0 3955176 3917272 0 0 0 0 0 0 0 0 1 0 0 4139 27264 5850 17 18<br>65<br> 0 0 0 3955176 3916520 0 0 0 0 0 0 0 0 1 0 0 4107 27259 5867 19 14<br>67<br><br>result of iostat<br> extended device statistics
<br> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device<br> 0.0 1.0 0.0 775.9 0.0 0.0 0.0 27.3 0 3 c1t0d0s3<br> 1.0 2.0 8.0 744.1 0.0 0.0 0.0 11.4 0 3 c1t0d0s3<br>
0.0 2.0 0.0 864.0 0.0 0.0 0.0 15.7 0 3 c1t0d0s3<br> 0.0 4.0 0.0 856.0 0.0 0.1 0.0 17.3 0 3 c1t0d0s3<br> 0.0 1.0 0.0 456.0 0.0 0.0 0.0 19.5 0 2 c1t0d0s3<br>
0.0 1.0 0.0 856.1 0.0 0.0 0.0 28.1 0 3 c1t0d0s3<br> 0.0 1.0 0.0 855.9 0.0 0.0 0.0 24.4 0 2 c1t0d0s3<br> 0.0 1.0 0.0 856.1 0.0 0.0 0.0 27.3 0 3 c1t0d0s3<br> 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0s3<br> 0.0 1.0 0.0 856.0 0.0 0.0 0.0 19.0 0 2 c1t0d0s3<br> 0.0 1.0 0.0 856.0 0.0 0.0 0.0 26.9 0 3 c1t0d0s3<br> 0.0
1.0 0.0 832.1 0.0 0.0 0.0 24.8 0 2 c1t0d0s3<br> 0.0 1.0 0.0 735.9 0.0 0.0 0.0 26.4 0 3 c1t0d0s3<br> 0.0 1.0 0.0 856.1 0.0 0.0 0.0 28.7 0 3 c1t0d0s3<br> 0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0s3<br><br>[Question]<br>Question 1.<br> Does the "fifo_size" in above message mean "log_fifo_size" in global<br> option ?<br> Is the value "fifo_size='1000'" max value ?
<br> Is it possible to set "log_fifo_size" far more than 1000 ?<br> Is it possible to set "sync" far more than 1000 ?<br> If possible, then how do I do it ?<br><br>Question 2.<br> To achieve the NEED, I am considering the parameters below,
<br> sync()<br> log_fifo_size()<br> .<br> Are there any other parameters I MUST consider for syslog-ng<br> configuration?<br><br>Regards<br><br>George<br>-------------- next part --------------<br>An HTML attachment was scrubbed...
<br>URL: <a href="http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070620/35731185/attachment.htm">http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070620/35731185/attachment.htm</a><br><br>------------------------------
<br><br>Message: 4<br>Date: Wed, 20 Jun 2007 08:03:20 +0100<br>From: "Geller, Sandor (IT)" <<a href="mailto:Sandor.Geller@morganstanley.com">Sandor.Geller@morganstanley.com</a>><br>Subject: RE: [syslog-ng] syslog-ng
2.0.4 How can syslog-ng achieve<br> thisperformance???<br>To: "Syslog-ng users' and developers' mailing list"<br> <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>
><br>Message-ID:<br> <<a href="mailto:14F0A35F6E466D48BF11108F4E09E68C05756F01@LNWEXMB58.msad.ms.com">14F0A35F6E466D48BF11108F4E09E68C05756F01@LNWEXMB58.msad.ms.com</a>><br>Content-Type: text/plain; charset="US-ASCII"
<br><br>> Hi All,<br>><br>> NEED HELP!!!!!<br><br>Don't panic :)<br><br>> [Environment]<br>> SunOS 5.9 Generic_122300-07 sun4u sparc SUNW<br>> syslog-ng 2.0.4<br>> disk : single disk (no RAID)<br>
> syslog-ng conf (global option part)<br>> sync (3000);<br>> time_reopen (10);<br>> time_sleep(0);<br>> log_fifo_size (10000);<br>> long_hostnames (off);
<br>> use_dns (no);<br>> use_fqdn (no);<br>> create_dirs (no);<br>> keep_hostname (yes);<br>><br>> [NEED]<br>> I want syslog-ng to write logs to local disk at pace of about 4000
<br>> lines per second without any lines losing.<br>> However, lines were in fact lost in the local file.<br><br>It would be good to see the statistics of syslog-ng. You should<br>set the stats_freq() option as well, and analyse the output. I
<br>would like to recommend using stats_freq(60);<br><br>As you have omitted your log sources I don't know whether you are<br>logging messages originating from the network. If you did, you<br>should check the receive buffer options.
<br><br>Also please note that using time_sleep(0) might cause performance<br>drops, so you should try using time_sleep(10) or higher, the optimal<br>setting depends on your environment...<br><br>> I am trying to know the syslog-ng max performable point.
<br><br>Depends on the speed of the CPU, the disks, your syslog-ng filters,<br>the ordering of the filters, ...<br><br>> [Consideration]<br>> "log_fifo_size" in global option is set as 10000.<br>> I tried to set sync() parameter, for instance 3000, in global
<br>> option section.<br>> This did not succeed with this messages when reloading the process,<br>> " The value of flush_lines must be less than fifo_size;<br>> fifo_size='1000',<br>> flush_lines='3000' ".
<br><br>Looks like line #502 of logwriter.c might be the cause of this.<br>Seems that the global log_fifo_size isn't propagated correctly.<br><br>However you can override that by using the log_fifo_size() option<br>in your destination definition too.
<br><br>Regards,<br><br>Sandor<br>--------------------------------------------------------<br><br>NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error.
<br><br><br>------------------------------<br><br>Message: 5<br>Date: Wed, 20 Jun 2007 10:47:50 +0200<br>From: Balazs Scheidler <<a href="mailto:bazsi@balabit.hu">bazsi@balabit.hu</a>><br>Subject: RE: [syslog-ng] syslog-ng
2.0.4 How can syslog-ng achieve<br> thisperformance???<br>To: Syslog-ng users' and developers' mailing list<br> <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>Message-ID: <
<a href="mailto:1182329270.6482.41.camel@bzorp.balabit">1182329270.6482.41.camel@bzorp.balabit</a>><br>Content-Type: text/plain<br><br>On Wed, 2007-06-20 at 08:03 +0100, Geller, Sandor (IT) wrote:<br>> > [NEED]<br>
> > I want syslog-ng to write logs to local disk at pace of about 4000<br>> > lines per second without any lines losing.<br>> > However, lines were in fact lost in the local file.<br>><br>> It would be good to see the statistics of syslog-ng. You should
<br>> set the stats_freq() option as well, and analyse the output. I<br>> would like to recommend using stats_freq(60);<br>><br>> As you have omitted your log sources I don't know whether you are<br>> logging messages originating from the network. If you did, you
<br>> should check the receive buffer options.<br>><br>> Also please note that using time_sleep(0) might cause performance<br>> drops, so you should try using time_sleep(10) or higher, the optimal<br>> setting depends on your environment...
<br>><br><br>First of all we need to know what your exact scenario is. You might be<br>missing a receive buffer size tweak, or you might have something else.<br>The information you provided is not enough.<br><br>> > [Consideration]
<br>> > "log_fifo_size" in global option is set as 10000.<br>> > I tried to set sync() parameter, for instance 3000, in global<br>> > option section.<br>> > This did not succeed with this messages when reloading the process,
<br>> > " The value of flush_lines must be less than fifo_size;<br>> > fifo_size='1000',<br>> > flush_lines='3000' ".<br>><br>> Looks like line #502 of logwriter.c might be the cause of this.
<br>> Seems that the global log_fifo_size isn't propagated correctly.<br>><br>> However you can override that by using the log_fifo_size() option<br>> in your destination definition too.<br><br>Right, the log_fifo_size() limit propagation has a problem, it maximizes
<br>the fifo size in 1000 entries, unless specified locally. This patch<br>fixes it:<br><br>diff --git a/src/logwriter.c b/src/logwriter.c<br>index eea6814..955c333 100644<br>--- a/src/logwriter.c<br>+++ b/src/logwriter.c
<br>@@ -499,7 +499,7 @@ log_writer_options_init(LogWriterOptions *options, GlobalConfig *cfg, guint32 fl<br> options->template = template;<br> options->flags = flags;<br> if (options->fifo_size == -1)<br>- options->fifo_size = MIN(1000, cfg->log_fifo_size);
<br>+ options->fifo_size = MAX(1000, cfg->log_fifo_size);<br> if (options->use_time_recvd == -1)<br> options->use_time_recvd = cfg->use_time_recvd;<br><br><br><br>--<br>Bazsi<br><br><br><br>------------------------------
<br><br>_______________________________________________<br>syslog-ng maillist - <a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a><br><a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br><br><br>End of syslog-ng Digest, Vol 26, Issue 18<br>*****************************************<br></blockquote></div><br>