puzzled about the system resources used by syslog-ng
Eenvironment : under linux 1 central log-collecting server.syslog-ng 2 client: syslog sending logs to central log-collecting server. And I have two central log-collecting servers which is logging for different clients,I call them A and B. configuration 1 The A ` configuration as follows: options { use_dns(no); create_dirs(yes);ts_format(iso); }; source src { udp(ip(0.0.0.0) port(514)); }; filter f_kern { facility(kern); }; filter f_authpriv { facility(auth,authpriv); }; filter f_mail { facility(mail); }; filter f_cron { facility(cron); }; filter f_boot { facility(local7); }; filter f_spooler { facility(uucp, news) and level(crit..emerg); }; filter f_messages { level(info..emerg) and not facility(authpriv, cron, mail); }; destination kern { file("/home/syslog-ng/$YEAR/$HOST/kernel-$MONTH"); }; destination authpriv { file("/home/syslog-ng/$YEAR/$HOST/secure-$MONTH"); }; destination mail { file("/home/syslog-ng/$YEAR/$HOST/maillog-$MONTH"); }; destination cron { file("/home/syslog-ng/$YEAR/$HOST/cron-$MONTH"); }; destination boot { file("/home/syslog-ng/$YEAR/$HOST/boot.log-$MONTH"); }; destination spooler { file("/home/syslog-ng/$YEAR/$HOST/spooler-$MONTH"); }; destination messages { file("/home/syslog-ng/$YEAR/$HOST/messages-$MONTH"); }; log { source(src); filter(f_kern); destination(kern); }; log { source(src); filter(f_authpriv); destination(authpriv); }; log { source(src); filter(f_mail); destination(mail); }; log { source(src); filter(f_cron); destination(cron); }; log { source(src); filter(f_boot); destination(boot); }; log { source(src); filter(f_spooler); destination(spooler); }; log { source(src); filter(f_messages); destination(messages); }; 2 The B` configuration as follows: source src { unix-stream("/dev/log");internal();udp(ip(0.0.0.0) port(514)); }; destination dsearch { file("/home/work/syslog-ng/log/$HOST"); }; log { source(src); destination(dsearch); }; QUESTIONS: A server log files: ps aux /home/syslog-ng/2007/ 10.11.100.3 / messages %MEM VSZ RSS 965 directories 3860 6.8 567772 562524 B server log files: ps aux /home/work/syslog-ng/log/10.12.55.6 % MEM VSZ RSS 4482 0.0 2232 1276 I think the MEM VSZ and RSS usage is related to the logfile numbers, A server has fewer logfiles than B server, while A server has higher usages of MEM VSZ and RSS than B server, I don`t know why ?
On Thu, 2007-09-20 at 12:27 +0800, liuruihong wrote:
Eenvironment :
under linux
1 central log-collecting server.syslog-ng
2 client: syslog sending logs to central log-collecting server.
And I have two central log-collecting servers which is logging for different clients,I call them A and B.
configuration
1 The A ` configuration as follows:
options { use_dns(no); create_dirs(yes);ts_format(iso); };
source src { udp(ip(0.0.0.0) port(514)); };
filter f_kern { facility(kern); };
filter f_authpriv { facility(auth,authpriv); };
filter f_mail { facility(mail); };
filter f_cron { facility(cron); };
filter f_boot { facility(local7); };
filter f_spooler { facility(uucp, news) and level(crit..emerg); };
filter f_messages { level(info..emerg) and not facility(authpriv, cron, mail); };
destination kern { file("/home/syslog-ng/$YEAR/$HOST/kernel-$MONTH"); };
destination authpriv { file("/home/syslog-ng/$YEAR/$HOST/secure-$MONTH"); };
destination mail { file("/home/syslog-ng/$YEAR/$HOST/maillog-$MONTH"); };
destination cron { file("/home/syslog-ng/$YEAR/$HOST/cron-$MONTH"); };
destination boot { file("/home/syslog-ng/$YEAR/$HOST/boot.log-$MONTH"); };
destination spooler { file("/home/syslog-ng/$YEAR/$HOST/spooler-$MONTH"); };
destination messages { file("/home/syslog-ng/$YEAR/$HOST/messages-$MONTH"); };
log { source(src); filter(f_kern); destination(kern); };
log { source(src); filter(f_authpriv); destination(authpriv); };
log { source(src); filter(f_mail); destination(mail); };
log { source(src); filter(f_cron); destination(cron); };
log { source(src); filter(f_boot); destination(boot); };
log { source(src); filter(f_spooler); destination(spooler); };
log { source(src); filter(f_messages); destination(messages); };
2 The B` configuration as follows:
source src { unix-stream("/dev/log");internal();udp(ip(0.0.0.0) port(514)); };
destination dsearch { file("/home/work/syslog-ng/log/$HOST"); };
log { source(src); destination(dsearch); };
QUESTIONS:
A server log files: ps aux
/home/syslog-ng/2007/ 10.11.100.3 / messages %MEM VSZ RSS
965 directories 3860 6.8 567772 562524
B server log files: ps aux
/home/work/syslog-ng/log/10.12.55.6 % MEM VSZ RSS
4482 0.0 2232 1276
I think the MEM VSZ and RSS usage is related to the logfile numbers, A server has fewer logfiles than B server, while A server has higher usages of MEM VSZ and RSS than B server,
I don`t know why ?
Hmm... it it a genuine leak, I mean does it continously grow, or it stays at this limit and gets decreased during off-peak time? In syslog-ng every destination file has its own buffer, though that's allocated on-demand, the maximum size can be specified using log_fifo_size(), though the minimum value is 1000 log messages. 1000 messages, 300 bytes on average is about 300kB maximum memory usage per file. How many messages are processed by syslog-ng here? Is each file receiving the same amount of messages? -- Bazsi
I see,perhaps the messages A received are different from B. -----邮件原件----- 发件人: syslog-ng-bounces@lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] 代表 Balazs Scheidler 发送时间: 2007年9月20日 14:38 收件人: Syslog-ng users' and developers' mailing list 主题: Re: [syslog-ng] puzzled about the system resources used by syslog-ng On Thu, 2007-09-20 at 12:27 +0800, liuruihong wrote:
Eenvironment :
under linux
1 central log-collecting server.syslog-ng
2 client: syslog sending logs to central log-collecting server.
And I have two central log-collecting servers which is logging for different clients,I call them A and B.
configuration
1 The A ` configuration as follows:
options { use_dns(no); create_dirs(yes);ts_format(iso); };
source src { udp(ip(0.0.0.0) port(514)); };
filter f_kern { facility(kern); };
filter f_authpriv { facility(auth,authpriv); };
filter f_mail { facility(mail); };
filter f_cron { facility(cron); };
filter f_boot { facility(local7); };
filter f_spooler { facility(uucp, news) and level(crit..emerg); };
filter f_messages { level(info..emerg) and not facility(authpriv, cron, mail); };
destination kern { file("/home/syslog-ng/$YEAR/$HOST/kernel-$MONTH"); };
destination authpriv { file("/home/syslog-ng/$YEAR/$HOST/secure-$MONTH"); };
destination mail { file("/home/syslog-ng/$YEAR/$HOST/maillog-$MONTH"); };
destination cron { file("/home/syslog-ng/$YEAR/$HOST/cron-$MONTH"); };
destination boot { file("/home/syslog-ng/$YEAR/$HOST/boot.log-$MONTH"); };
destination spooler { file("/home/syslog-ng/$YEAR/$HOST/spooler-$MONTH"); };
destination messages { file("/home/syslog-ng/$YEAR/$HOST/messages-$MONTH"); };
log { source(src); filter(f_kern); destination(kern); };
log { source(src); filter(f_authpriv); destination(authpriv); };
log { source(src); filter(f_mail); destination(mail); };
log { source(src); filter(f_cron); destination(cron); };
log { source(src); filter(f_boot); destination(boot); };
log { source(src); filter(f_spooler); destination(spooler); };
log { source(src); filter(f_messages); destination(messages); };
2 The B` configuration as follows:
source src { unix-stream("/dev/log");internal();udp(ip(0.0.0.0) port(514)); };
destination dsearch { file("/home/work/syslog-ng/log/$HOST"); };
log { source(src); destination(dsearch); };
QUESTIONS:
A server log files: ps aux
/home/syslog-ng/2007/ 10.11.100.3 / messages %MEM VSZ RSS
965 directories 3860 6.8 567772 562524
B server log files: ps aux
/home/work/syslog-ng/log/10.12.55.6 % MEM VSZ RSS
4482 0.0 2232 1276
I think the MEM VSZ and RSS usage is related to the logfile numbers, A server has fewer logfiles than B server, while A server has higher usages of MEM VSZ and RSS than B server,
I don`t know why ?
Hmm... it it a genuine leak, I mean does it continously grow, or it stays at this limit and gets decreased during off-peak time? In syslog-ng every destination file has its own buffer, though that's allocated on-demand, the maximum size can be specified using log_fifo_size(), though the minimum value is 1000 log messages. 1000 messages, 300 bytes on average is about 300kB maximum memory usage per file. How many messages are processed by syslog-ng here? Is each file receiving the same amount of messages? -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
participants (2)
-
Balazs Scheidler
-
liuruihong