[syslog-ng] RE: syslog-ng 2.0.4 How can syslog-ng achieve this
performance???
Root Administrator
root.regist at gmail.com
Mon Jun 25 05:10:32 CEST 2007
Geller, Sandor
Balazs Scheidler
Thanks 4 ur advise.
Please pardon me for my short information.
The scenario is this,
-----
There is a network device which sends logs to syslog server over network at
a pace of approx 4000 logs/sec.
Syslog server has its own mission to handle them.
In case, syslog server must write logs to local disk, the server in fact
drops some logs in the local file.
Consequently, I turned to think of tuning some tweak in syslog-ng parameter
or kernel parameter.
I have tuned kernel parameter by setting "udp_recv_hiwat" to its maximum
value, and udp_max_buf to the value of 300 times of the default value.
And, I have come up with this idea.
1. Increase "sync" parameter to buffer some logs and write logs , not at
the pace of every second.
I tried to increase sync as well as log_fifo_size.
First, I set sync as 3000 , log_fifo_size as 10000.
However, it was not liked, with the message
" The value of flush_lines must be less than fifo_size;
fifo_size='1000',flush_lines='3000' ".
Syslog-ng is configured as follows;
sync (3000);
time_reopen (10);
time_sleep(0);
log_fifo_size (10000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
source s_test { udp(ip(0.0.0.0) port(514)); };
destination d_local4 { file("/var/log/local4"); };
filter f_local4_al { facility(local4) and level(info) and
match("xxxxxx") and filter(test); };
filter test { match("10600[1267]") or match("10601[0-8]") or match
("10602[0124567]") or match("106100") or
match("20900[345]") or match("500004"); };
log { source(s_test); filter(f_local4_al); destination(d_local4); };
Any advice about how sync works , and how log_fifo_size works will
greatly help.
Is there any other way than editing the logwriter.c file and
re-compile it?
Is there difference between setting sync and log_fifo_size in global
option section and individual destination section
in terms of its effect?
2. "fsync" parameter would be thouhgt as second chance to overcome this
problem.
However, no userful information cound not be found on web.
I set fsync in destination section, but it was rejected when reloading
the process.
Any advice about how fsync works will greatly help.
In addition, how can I get STATS information of syslog-ng?
I have added "stats_freq (60);" in global option section, but I could not
get any information in /var/adm/messages.
Where does syslog-ng output the stats information?
Thanks!
Regards
George
------------------------------
>
> Message: 3
> Date: Wed, 20 Jun 2007 11:12:40 +0900
> From: "Root Administrator" <root.regist at gmail.com>
> Subject: [syslog-ng] syslog-ng 2.0.4 How can syslog-ng achieve this
> performance???
> To: syslog-ng at lists.balabit.hu
> Message-ID:
> <8ee7d730706191912l357ecabfgb137b7297c1bfbf0 at mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Hi All,
>
> NEED HELP!!!!!
>
> [Environment]
> SunOS 5.9 Generic_122300-07 sun4u sparc SUNW
> syslog-ng 2.0.4
> disk : single disk (no RAID)
> syslog-ng conf (global option part)
> sync (3000);
> time_reopen (10);
> time_sleep(0);
> log_fifo_size (10000);
> long_hostnames (off);
> use_dns (no);
> use_fqdn (no);
> create_dirs (no);
> keep_hostname (yes);
>
> [NEED]
> I want syslog-ng to write logs to local disk at pace of about 4000
> lines per second without any lines losing.
> However, lines were in fact lost in the local file.
> I am trying to know the syslog-ng max performable point.
>
> [Consideration]
> "log_fifo_size" in global option is set as 10000.
> I tried to set sync() parameter, for instance 3000, in global option
> section.
> This did not succeed with this messages when reloading the process,
> " The value of flush_lines must be less than fifo_size; fifo_size='1000',
> flush_lines='3000' ".
> The administration guide says "sync" is alias for "flush_lines".
>
> In addition, resource usage at 4000lines/s load was as below,
>
> result of vmstat
> kthr memory page disk faults cpu
> r b w swap free re mf pi po fr de sr s3 sd sd -- in sy cs us sy
> id
> 0 0 0 3954792 3895288 155 8 353 0 0 0 12 0 2 0 0 669 4368 857 4 3
> 93
> 0 0 0 3955248 3926424 3 8 0 0 0 0 0 0 2 0 0 4112 27017 5889 21
> 15
> 64
> 0 0 0 3955248 3925664 0 0 0 0 0 0 0 0 3 0 0 4120 27251 5653 16
> 19
> 64
> 0 0 0 3955248 3924528 0 0 0 0 0 0 0 0 1 0 0 4129 27251 5914 15
> 18
> 67
> 0 0 0 3955248 3923400 0 0 0 0 0 0 0 0 1 0 0 4113 27236 6052 17
> 15
> 68
> 0 0 0 3955176 3922568 0 0 0 0 0 0 0 0 32 0 0 4156 26028 5405 19
> 13
> 68
> 0 0 0 3955176 3921808 0 0 0 0 0 0 0 0 1 0 0 4161 27316 5757 16
> 18
> 66
> 0 0 0 3955176 3921056 0 0 0 0 0 0 0 0 1 0 0 4120 27254 6136 14
> 18
> 68
> 0 0 0 3955176 3920296 0 0 0 0 0 0 0 0 1 0 0 4110 27244 5648 18
> 17
> 65
> 0 0 0 3955176 3919544 0 0 0 0 0 0 0 0 1 0 0 4115 27253 6042 17
> 19
> 64
> 0 0 0 3955176 3918784 0 0 0 0 0 0 0 0 1 0 0 4108 27238 6469 18
> 16
> 65
> 0 0 0 3955176 3918032 0 0 0 0 0 0 0 0 1 0 0 4107 27235 6106 16
> 18
> 66
> 0 0 0 3955176 3917272 0 0 0 0 0 0 0 0 1 0 0 4139 27264 5850 17
> 18
> 65
> 0 0 0 3955176 3916520 0 0 0 0 0 0 0 0 1 0 0 4107 27259 5867 19
> 14
> 67
>
> result of iostat
> extended device statistics
> r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
> 0.0 1.0 0.0 775.9 0.0 0.0 0.0 27.3 0 3 c1t0d0s3
> 1.0 2.0 8.0 744.1 0.0 0.0 0.0 11.4 0 3 c1t0d0s3
> 0.0 2.0 0.0 864.0 0.0 0.0 0.0 15.7 0 3 c1t0d0s3
> 0.0 4.0 0.0 856.0 0.0 0.1 0.0 17.3 0 3 c1t0d0s3
> 0.0 1.0 0.0 456.0 0.0 0.0 0.0 19.5 0 2 c1t0d0s3
> 0.0 1.0 0.0 856.1 0.0 0.0 0.0 28.1 0 3 c1t0d0s3
> 0.0 1.0 0.0 855.9 0.0 0.0 0.0 24.4 0 2 c1t0d0s3
> 0.0 1.0 0.0 856.1 0.0 0.0 0.0 27.3 0 3 c1t0d0s3
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0s3
> 0.0 1.0 0.0 856.0 0.0 0.0 0.0 19.0 0 2 c1t0d0s3
> 0.0 1.0 0.0 856.0 0.0 0.0 0.0 26.9 0 3 c1t0d0s3
> 0.0 1.0 0.0 832.1 0.0 0.0 0.0 24.8 0 2 c1t0d0s3
> 0.0 1.0 0.0 735.9 0.0 0.0 0.0 26.4 0 3 c1t0d0s3
> 0.0 1.0 0.0 856.1 0.0 0.0 0.0 28.7 0 3 c1t0d0s3
> 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0s3
>
> [Question]
> Question 1.
> Does the "fifo_size" in above message mean "log_fifo_size" in global
> option ?
> Is the value "fifo_size='1000'" max value ?
> Is it possible to set "log_fifo_size" far more than 1000 ?
> Is it possible to set "sync" far more than 1000 ?
> If possible, then how do I do it ?
>
> Question 2.
> To achieve the NEED, I am considering the parameters below,
> sync()
> log_fifo_size()
> .
> Are there any other parameters I MUST consider for syslog-ng
> configuration?
>
> Regards
>
> George
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL:
> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070620/35731185/attachment.htm
>
> ------------------------------
>
> Message: 4
> Date: Wed, 20 Jun 2007 08:03:20 +0100
> From: "Geller, Sandor (IT)" <Sandor.Geller at morganstanley.com>
> Subject: RE: [syslog-ng] syslog-ng 2.0.4 How can syslog-ng achieve
> thisperformance???
> To: "Syslog-ng users' and developers' mailing list"
> <syslog-ng at lists.balabit.hu>
> Message-ID:
> <14F0A35F6E466D48BF11108F4E09E68C05756F01 at LNWEXMB58.msad.ms.com>
> Content-Type: text/plain; charset="US-ASCII"
>
> > Hi All,
> >
> > NEED HELP!!!!!
>
> Don't panic :)
>
> > [Environment]
> > SunOS 5.9 Generic_122300-07 sun4u sparc SUNW
> > syslog-ng 2.0.4
> > disk : single disk (no RAID)
> > syslog-ng conf (global option part)
> > sync (3000);
> > time_reopen (10);
> > time_sleep(0);
> > log_fifo_size (10000);
> > long_hostnames (off);
> > use_dns (no);
> > use_fqdn (no);
> > create_dirs (no);
> > keep_hostname (yes);
> >
> > [NEED]
> > I want syslog-ng to write logs to local disk at pace of about 4000
> > lines per second without any lines losing.
> > However, lines were in fact lost in the local file.
>
> It would be good to see the statistics of syslog-ng. You should
> set the stats_freq() option as well, and analyse the output. I
> would like to recommend using stats_freq(60);
>
> As you have omitted your log sources I don't know whether you are
> logging messages originating from the network. If you did, you
> should check the receive buffer options.
>
> Also please note that using time_sleep(0) might cause performance
> drops, so you should try using time_sleep(10) or higher, the optimal
> setting depends on your environment...
>
> > I am trying to know the syslog-ng max performable point.
>
> Depends on the speed of the CPU, the disks, your syslog-ng filters,
> the ordering of the filters, ...
>
> > [Consideration]
> > "log_fifo_size" in global option is set as 10000.
> > I tried to set sync() parameter, for instance 3000, in global
> > option section.
> > This did not succeed with this messages when reloading the process,
> > " The value of flush_lines must be less than fifo_size;
> > fifo_size='1000',
> > flush_lines='3000' ".
>
> Looks like line #502 of logwriter.c might be the cause of this.
> Seems that the global log_fifo_size isn't propagated correctly.
>
> However you can override that by using the log_fifo_size() option
> in your destination definition too.
>
> Regards,
>
> Sandor
> --------------------------------------------------------
>
> NOTICE: If received in error, please destroy and notify sender. Sender
> does not intend to waive confidentiality or privilege. Use of this email is
> prohibited when received in error.
>
>
> ------------------------------
>
> Message: 5
> Date: Wed, 20 Jun 2007 10:47:50 +0200
> From: Balazs Scheidler <bazsi at balabit.hu>
> Subject: RE: [syslog-ng] syslog-ng 2.0.4 How can syslog-ng achieve
> thisperformance???
> To: Syslog-ng users' and developers' mailing list
> <syslog-ng at lists.balabit.hu>
> Message-ID: <1182329270.6482.41.camel at bzorp.balabit>
> Content-Type: text/plain
>
> On Wed, 2007-06-20 at 08:03 +0100, Geller, Sandor (IT) wrote:
> > > [NEED]
> > > I want syslog-ng to write logs to local disk at pace of about 4000
> > > lines per second without any lines losing.
> > > However, lines were in fact lost in the local file.
> >
> > It would be good to see the statistics of syslog-ng. You should
> > set the stats_freq() option as well, and analyse the output. I
> > would like to recommend using stats_freq(60);
> >
> > As you have omitted your log sources I don't know whether you are
> > logging messages originating from the network. If you did, you
> > should check the receive buffer options.
> >
> > Also please note that using time_sleep(0) might cause performance
> > drops, so you should try using time_sleep(10) or higher, the optimal
> > setting depends on your environment...
> >
>
> First of all we need to know what your exact scenario is. You might be
> missing a receive buffer size tweak, or you might have something else.
> The information you provided is not enough.
>
> > > [Consideration]
> > > "log_fifo_size" in global option is set as 10000.
> > > I tried to set sync() parameter, for instance 3000, in global
> > > option section.
> > > This did not succeed with this messages when reloading the process,
> > > " The value of flush_lines must be less than fifo_size;
> > > fifo_size='1000',
> > > flush_lines='3000' ".
> >
> > Looks like line #502 of logwriter.c might be the cause of this.
> > Seems that the global log_fifo_size isn't propagated correctly.
> >
> > However you can override that by using the log_fifo_size() option
> > in your destination definition too.
>
> Right, the log_fifo_size() limit propagation has a problem, it maximizes
> the fifo size in 1000 entries, unless specified locally. This patch
> fixes it:
>
> diff --git a/src/logwriter.c b/src/logwriter.c
> index eea6814..955c333 100644
> --- a/src/logwriter.c
> +++ b/src/logwriter.c
> @@ -499,7 +499,7 @@ log_writer_options_init(LogWriterOptions *options,
> GlobalConfig *cfg, guint32 fl
> options->template = template;
> options->flags = flags;
> if (options->fifo_size == -1)
> - options->fifo_size = MIN(1000, cfg->log_fifo_size);
> + options->fifo_size = MAX(1000, cfg->log_fifo_size);
> if (options->use_time_recvd == -1)
> options->use_time_recvd = cfg->use_time_recvd;
>
>
>
> --
> Bazsi
>
>
>
> ------------------------------
>
> _______________________________________________
> syslog-ng maillist - syslog-ng at lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>
>
> End of syslog-ng Digest, Vol 26, Issue 18
> *****************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070625/f248fa82/attachment-0001.html
More information about the syslog-ng
mailing list