Geller, Sandor
Balazs Scheidler
Thanks 4 ur advise.
Please pardon me for my short information.
The scenario is this,
-----
There is a network device which sends logs to syslog server over network at a pace of approx 4000 logs/sec.
Syslog server has its own mission to handle them.
In case, syslog server must write logs to local disk, the server in fact drops some logs in the local file.
Consequently, I turned to think of tuning some tweak in syslog-ng parameter or kernel parameter.
I have tuned kernel parameter by setting "udp_recv_hiwat" to its maximum value, and udp_max_buf to the value of 300 times of the default value.
And, I have come up with this idea.
1. Increase "sync" parameter to buffer some logs and write logs , not at the pace of every second.
I tried to increase sync as well as log_fifo_size.
First, I set sync as 3000 , log_fifo_size as 10000.
However, it was not liked, with the message
" The value of flush_lines must be less than fifo_size; fifo_size='1000',flush_lines='3000' ".
Syslog-ng is configured as follows;
sync (3000);
time_reopen (10);
time_sleep(0);
log_fifo_size (10000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
source s_test { udp(ip(
0.0.0.0) port(514)); };
destination d_local4 { file("/var/log/local4"); };
filter f_local4_al { facility(local4) and level(info) and match("xxxxxx") and filter(test); };
filter test { match("10600[1267]") or match("10601[0-8]") or match ("10602[0124567]") or match("106100") or
match("20900[345]") or match("500004"); };
log { source(s_test); filter(f_local4_al); destination(d_local4); };
Any advice about how sync works , and how log_fifo_size works will greatly help.
Is there any other way than editing the logwriter.c file and re-compile it?
Is there difference between setting sync and log_fifo_size in global option section and individual destination section
in terms of its effect?
2. "fsync" parameter would be thouhgt as second chance to overcome this problem.
However, no userful information cound not be found on web.
I set fsync in destination section, but it was rejected when reloading the process.
Any advice about how fsync works will greatly help.
In addition, how can I get STATS information of syslog-ng?
I have added "stats_freq (60);" in global option section, but I could not get any information in /var/adm/messages.
Where does syslog-ng output the stats information?
Thanks!
Regards
George
------------------------------
Message: 3
Date: Wed, 20 Jun 2007 11:12:40 +0900
From: "Root Administrator" <root.regist@gmail.com>
Subject: [syslog-ng] syslog-ng
2.0.4 How can syslog-ng achieve this
performance???
To: syslog-ng@lists.balabit.hu
Message-ID:
<
8ee7d730706191912l357ecabfgb137b7297c1bfbf0@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi All,
NEED HELP!!!!!
[Environment]
SunOS 5.9 Generic_122300-07 sun4u sparc SUNW
syslog-ng 2.0.4
disk : single disk (no RAID)
syslog-ng conf (global option part)
sync (3000);
time_reopen (10);
time_sleep(0);
log_fifo_size (10000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (no);
keep_hostname (yes);
[NEED]
I want syslog-ng to write logs to local disk at pace of about 4000
lines per second without any lines losing.
However, lines were in fact lost in the local file.
I am trying to know the syslog-ng max performable point.
[Consideration]
"log_fifo_size" in global option is set as 10000.
I tried to set sync() parameter, for instance 3000, in global option
section.
This did not succeed with this messages when reloading the process,
" The value of flush_lines must be less than fifo_size; fifo_size='1000',
flush_lines='3000' ".
The administration guide says "sync" is alias for "flush_lines".
In addition, resource usage at 4000lines/s load was as below,
result of vmstat
kthr memory page disk faults cpu
r b w swap free re mf pi po fr de sr s3 sd sd -- in sy cs us sy
id
0 0 0 3954792 3895288 155 8 353 0 0 0 12 0 2 0 0 669 4368 857 4 3
93
0 0 0 3955248 3926424 3 8 0 0 0 0 0 0 2 0 0 4112 27017 5889 21 15
64
0 0 0 3955248 3925664 0 0 0 0 0 0 0 0 3 0 0 4120 27251 5653 16 19
64
0 0 0 3955248 3924528 0 0 0 0 0 0 0 0 1 0 0 4129 27251 5914 15 18
67
0 0 0 3955248 3923400 0 0 0 0 0 0 0 0 1 0 0 4113 27236 6052 17 15
68
0 0 0 3955176 3922568 0 0 0 0 0 0 0 0 32 0 0 4156 26028 5405 19 13
68
0 0 0 3955176 3921808 0 0 0 0 0 0 0 0 1 0 0 4161 27316 5757 16 18
66
0 0 0 3955176 3921056 0 0 0 0 0 0 0 0 1 0 0 4120 27254 6136 14 18
68
0 0 0 3955176 3920296 0 0 0 0 0 0 0 0 1 0 0 4110 27244 5648 18 17
65
0 0 0 3955176 3919544 0 0 0 0 0 0 0 0 1 0 0 4115 27253 6042 17 19
64
0 0 0 3955176 3918784 0 0 0 0 0 0 0 0 1 0 0 4108 27238 6469 18 16
65
0 0 0 3955176 3918032 0 0 0 0 0 0 0 0 1 0 0 4107 27235 6106 16 18
66
0 0 0 3955176 3917272 0 0 0 0 0 0 0 0 1 0 0 4139 27264 5850 17 18
65
0 0 0 3955176 3916520 0 0 0 0 0 0 0 0 1 0 0 4107 27259 5867 19 14
67
result of iostat
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
0.0 1.0 0.0 775.9 0.0 0.0 0.0 27.3 0 3 c1t0d0s3
1.0 2.0 8.0 744.1 0.0 0.0 0.0 11.4 0 3 c1t0d0s3
0.0 2.0 0.0 864.0 0.0 0.0 0.0 15.7 0 3 c1t0d0s3
0.0 4.0 0.0 856.0 0.0 0.1 0.0 17.3 0 3 c1t0d0s3
0.0 1.0 0.0 456.0 0.0 0.0 0.0 19.5 0 2 c1t0d0s3
0.0 1.0 0.0 856.1 0.0 0.0 0.0 28.1 0 3 c1t0d0s3
0.0 1.0 0.0 855.9 0.0 0.0 0.0 24.4 0 2 c1t0d0s3
0.0 1.0 0.0 856.1 0.0 0.0 0.0 27.3 0 3 c1t0d0s3
0.0
0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0s3
0.0 1.0 0.0 856.0 0.0 0.0 0.0 19.0 0 2 c1t0d0s3
0.0 1.0 0.0 856.0 0.0 0.0 0.0 26.9 0 3 c1t0d0s3
0.0
1.0 0.0 832.1 0.0 0.0 0.0 24.8 0 2 c1t0d0s3
0.0 1.0 0.0 735.9 0.0 0.0 0.0 26.4 0 3 c1t0d0s3
0.0 1.0 0.0 856.1 0.0 0.0 0.0 28.7 0 3 c1t0d0s3
0.0 0.0
0.0 0.0 0.0 0.0 0.0 0.0 0 0 c1t0d0s3
[Question]
Question 1.
Does the "fifo_size" in above message mean "log_fifo_size" in global
option ?
Is the value "fifo_size='1000'" max value ?
Is it possible to set "log_fifo_size" far more than 1000 ?
Is it possible to set "sync" far more than 1000 ?
If possible, then how do I do it ?
Question 2.
To achieve the NEED, I am considering the parameters below,
sync()
log_fifo_size()
.
Are there any other parameters I MUST consider for syslog-ng
configuration?
Regards
George
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20070620/35731185/attachment.htm
------------------------------
Message: 4
Date: Wed, 20 Jun 2007 08:03:20 +0100
From: "Geller, Sandor (IT)" <Sandor.Geller@morganstanley.com>
Subject: RE: [syslog-ng] syslog-ng
2.0.4 How can syslog-ng achieve
thisperformance???
To: "Syslog-ng users' and developers' mailing list"
<syslog-ng@lists.balabit.hu
>
Message-ID:
<14F0A35F6E466D48BF11108F4E09E68C05756F01@LNWEXMB58.msad.ms.com>
Content-Type: text/plain; charset="US-ASCII"
> Hi All,
>
> NEED HELP!!!!!
Don't panic :)
> [Environment]
> SunOS 5.9 Generic_122300-07 sun4u sparc SUNW
> syslog-ng 2.0.4
> disk : single disk (no RAID)
> syslog-ng conf (global option part)
> sync (3000);
> time_reopen (10);
> time_sleep(0);
> log_fifo_size (10000);
> long_hostnames (off);
> use_dns (no);
> use_fqdn (no);
> create_dirs (no);
> keep_hostname (yes);
>
> [NEED]
> I want syslog-ng to write logs to local disk at pace of about 4000
> lines per second without any lines losing.
> However, lines were in fact lost in the local file.
It would be good to see the statistics of syslog-ng. You should
set the stats_freq() option as well, and analyse the output. I
would like to recommend using stats_freq(60);
As you have omitted your log sources I don't know whether you are
logging messages originating from the network. If you did, you
should check the receive buffer options.
Also please note that using time_sleep(0) might cause performance
drops, so you should try using time_sleep(10) or higher, the optimal
setting depends on your environment...
> I am trying to know the syslog-ng max performable point.
Depends on the speed of the CPU, the disks, your syslog-ng filters,
the ordering of the filters, ...
> [Consideration]
> "log_fifo_size" in global option is set as 10000.
> I tried to set sync() parameter, for instance 3000, in global
> option section.
> This did not succeed with this messages when reloading the process,
> " The value of flush_lines must be less than fifo_size;
> fifo_size='1000',
> flush_lines='3000' ".
Looks like line #502 of logwriter.c might be the cause of this.
Seems that the global log_fifo_size isn't propagated correctly.
However you can override that by using the log_fifo_size() option
in your destination definition too.
Regards,
Sandor
--------------------------------------------------------
NOTICE: If received in error, please destroy and notify sender. Sender does not intend to waive confidentiality or privilege. Use of this email is prohibited when received in error.
------------------------------
Message: 5
Date: Wed, 20 Jun 2007 10:47:50 +0200
From: Balazs Scheidler <bazsi@balabit.hu>
Subject: RE: [syslog-ng] syslog-ng
2.0.4 How can syslog-ng achieve
thisperformance???
To: Syslog-ng users' and developers' mailing list
<syslog-ng@lists.balabit.hu>
Message-ID: <
1182329270.6482.41.camel@bzorp.balabit>
Content-Type: text/plain
On Wed, 2007-06-20 at 08:03 +0100, Geller, Sandor (IT) wrote:
> > [NEED]
> > I want syslog-ng to write logs to local disk at pace of about 4000
> > lines per second without any lines losing.
> > However, lines were in fact lost in the local file.
>
> It would be good to see the statistics of syslog-ng. You should
> set the stats_freq() option as well, and analyse the output. I
> would like to recommend using stats_freq(60);
>
> As you have omitted your log sources I don't know whether you are
> logging messages originating from the network. If you did, you
> should check the receive buffer options.
>
> Also please note that using time_sleep(0) might cause performance
> drops, so you should try using time_sleep(10) or higher, the optimal
> setting depends on your environment...
>
First of all we need to know what your exact scenario is. You might be
missing a receive buffer size tweak, or you might have something else.
The information you provided is not enough.
> > [Consideration]
> > "log_fifo_size" in global option is set as 10000.
> > I tried to set sync() parameter, for instance 3000, in global
> > option section.
> > This did not succeed with this messages when reloading the process,
> > " The value of flush_lines must be less than fifo_size;
> > fifo_size='1000',
> > flush_lines='3000' ".
>
> Looks like line #502 of logwriter.c might be the cause of this.
> Seems that the global log_fifo_size isn't propagated correctly.
>
> However you can override that by using the log_fifo_size() option
> in your destination definition too.
Right, the log_fifo_size() limit propagation has a problem, it maximizes
the fifo size in 1000 entries, unless specified locally. This patch
fixes it:
diff --git a/src/logwriter.c b/src/logwriter.c
index eea6814..955c333 100644
--- a/src/logwriter.c
+++ b/src/logwriter.c
@@ -499,7 +499,7 @@ log_writer_options_init(LogWriterOptions *options, GlobalConfig *cfg, guint32 fl
options->template = template;
options->flags = flags;
if (options->fifo_size == -1)
- options->fifo_size = MIN(1000, cfg->log_fifo_size);
+ options->fifo_size = MAX(1000, cfg->log_fifo_size);
if (options->use_time_recvd == -1)
options->use_time_recvd = cfg->use_time_recvd;
--
Bazsi
------------------------------
_______________________________________________
syslog-ng maillist - syslog-ng@lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
End of syslog-ng Digest, Vol 26, Issue 18
*****************************************