[syslog-ng]syslog-ng performance problems
Stefan Wenger
syslog-ng@lists.balabit.hu
Sun, 5 Sep 2004 00:41:49 +0200
Hallo
Today I found the solution for the syslog-ng performance problem.
When I wanted to configure syslog-ng to forward all messages to
a syslog-ng test server, I saw in the /etc/hosts file that the=20
loghost was the same machine like the syslog-ng server.
So the syslog-ng server sent the same message to itself again
and again. #-( For some reason the traditional syslogd didn't do
that.
After I removed the loghost destination in the syslog-ng
config syslog-ng used between 3% and 6% of the CPU. Syslog-ng
is running on a machine with a 500 MHz 1 UltraSPARC IIe CPU and
there are sometimes about 300 Messages per second and more!
I think these are really acceptable values :-D
Thanks for your help
Stefan
-----Urspr=FCngliche Nachricht-----
Von: Balazs Scheidler [mailto:bazsi@balabit.hu]=20
Gesendet: Freitag, 3. September 2004 12:13
An: syslog-ng@lists.balabit.hu
Betreff: Re: [syslog-ng]syslog-ng performance problems
You have a lot of log() statements however I can't see right now why
syslog-ng is eating so much CPU time. Can you do some kind of profiling?
(oprofile with syslog-ng debug symbols, or easier compiling syslog-ng
with -pg and processing gmon.out with gprof)?
On Thu, 2004-09-02 at 14:22, west@x-dot.ch wrote:
> Hallo
>=20
> I have replaced the original syslog with syslog-ng.
> everything works fine. the only problem I have is that
> syslog-ng uses between 50% and 70% of the cpu.
> I think that it is a configuration problem because the
> old syslogd never uses more than 10% of the cpu.=20
>=20
> thank you for your help
> stefan
>=20
> here are my configuration files:
>=20
> traditional syslog:
>=20
> # /etc/syslog.conf syslog configuration
> file.###################################
>=20
> # output to local file "messages" for automatic log file analysis
> *.err;auth,daemon,mark,kern.debug;mail,user.notice =
/var/adm/messages
>=20
> # display emergencies on all terminals (uses WALL)
> *.emerg *
>=20
> #print time on console every 20mins (not needed if you have contool)
> #mark.* /dev/console
>=20
> kern.info ifdef(`LOGHOST', /var/log/kernlog, @loghost)
> user.info ifdef(`LOGHOST', /var/log/userlog, @loghost)
> mail.info ifdef(`LOGHOST', /var/log/maillog, @loghost)
> daemon.info ifdef(`LOGHOST', /var/log/daemonlog, @loghost)
> auth.info ifdef(`LOGHOST', /var/log/authlog, @loghost)
> lpr.info ifdef(`LOGHOST', /var/log/lprlog, @loghost)
> news,uucp.info ifdef(`LOGHOST', /var/log/newslog, @loghost)
> cron.info ifdef(`LOGHOST', /var/log/cronlog, @loghost)
>=20
> ## other "local" messages not yet used
> local0,local1.info ifdef(`LOGHOST', /var/log/local0log,
@loghost)
> local2,local3,local4.info ifdef(`LOGHOST', /var/log/local2log,
@loghost)
> local5.info ifdef(`LOGHOST', /var/log/local5log,
@loghost)
> local6.info ifdef(`LOGHOST', /var/log/local6log,
@loghost)
> local7.info ifdef(`LOGHOST', /var/log/local7log,
@loghost)
>=20
> # Put all alerts (& higher) into a seperate log:
> *.err ifdef(`LOGHOST', /var/log/alertlog, @loghost)
>
#########################################################################=
##
>=20
> syslog-ng:
> #
> # Syslog-ng configuration for SUN Solaris
> #
> # Copyright (c) 1999 anonymous
> # Copyright (c) 1999 Balazs Scheidler
> # Copyleft 2004 Stefan Wenger
> # $Id: syslog-ng.conf.sample,v 1.2 1999/11/15 12:30:41 bazsi Exp $
> #
> # Syslog-ng configuration file, compatible with default Debian syslogd
> # installation.
> #
>=20
> options {
> keep_hostname(yes);
> time_reopen (1);
> time_reap(300);
>=20
> use_dns(yes);
> use_fqdn(no);
> use_time_recvd(yes);
>=20
> dns_cache(yes);
> dns_cache_expire(3600);
> dns_cache_expire_failed(10);
>=20
> sync(4);
> gc_idle_threshold(300);
> gc_busy_threshold(1000);
> log_fifo_size(16777216);
> log_msg_size(8192);
> chain_hostnames(no);
>=20
> owner(root);
> group(nobody);
> perm(0644);
> dir_perm(0755);
> create_dirs(yes);
> };
>=20
> source src { sun_streams("/dev/log" door("/etc/.syslog_door")); =
internal
(); };
> source net { udp(); };
>=20
> destination alertlog { file("/var/log/alertlog"); };
> destination messages { file("/var/adm/messages"); };
> destination console { usertty("root"); };
> destination console_all { file("/dev/tty12"); };
>=20
> destination kernlog { file("/var/log/kernlog"); };
> destination userlog { file("/var/log/userlog"); };
> destination maillog { file("/var/log/maillog"); };
> destination daemonlog { file("/var/log/daemonlog"); };
> destination authlog { file("/var/log/authlog"); };
> destination lprlog { file("/var/log/lprlog"); };
> destination newslog { file("/var/log/newslog"); };
> destination cronlog { file("/var/log/cronlog"); };
>=20
> destination local0log { file("/var/log/local0log"); };
> destination local2log { file("/var/log/local2log"); };
> destination local5log { file("/var/log/local5log"); };
> destination local6log { file("/var/log/local6log"); };
> destination local7log { file("/var/log/local7log"); };
>=20
> destination fallbacklog { file("/var/log/fallbacklog"); };
>=20
> destination loghost { udp("loghost"); };
> #destination xconsole { pipe("/dev/xconsole"); };
>=20
> destination d_mysql { pipe("/tmp/mysql.pipe"
> template("INSERT INTO logs (host, facility, priority, level, tag,
date,time,
> program, msg)
> VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL',
> '$TAG','$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC',
> '$PROGRAM', '$MSG' );\n") template-escape(yes));
> };
>=20
> filter f_emerg { level(emerg); };
> filter f_kern_info {facility(kern) and not priority(debug); };
> filter f_user_info {facility(user) and not priority(debug); };
> filter f_mail_info {facility(mail) and not priority(debug); };
> filter f_daemon_info {facility(daemon) and not priority(debug); };
> filter f_auth_info {facility(auth) and not priority(debug); };
> filter f_lpr_info {facility(lpr) and not priority(debug); };
> filter f_news_info {(facility(news) or facility(uucp)) and not
> priority(debug);
> };
> filter f_cron_info {facility(cron) and not priority(debug); };
>=20
> filter f_local0 {facility(local0, local1) and not priority(debug); };
> filter f_local2 {facility(local2, local3, local4) and not =
priority(debug);
};
> filter f_local5 {facility(local5) and not priority(debug); };
> filter f_local6 {facility(local6) and not priority(debug); };
> filter f_local7 {facility(local7) and not priority(debug); };
> filter f_err {priority(err..emerg); };
> filter f_messages {priority(err..emerg) or facility(auth, daemon, =
kern)
> or (facility(mail, user) and priority(notice..emerg)); };
>=20
> log { source(src); filter(f_messages); destination(messages); };
> log { source(src); filter(f_emerg); destination(console); };
> log { source(src); filter(f_kern_info); destination(kernlog);
> destination(loghost); };
> log { source(src); filter(f_user_info); destination(userlog);
> destination(loghost); };
> log { source(src); filter(f_mail_info); destination(maillog);
> destination(loghost); };
> log { source(src); filter(f_daemon_info); destination(daemonlog);
> destination(loghost); };
> log { source(src); filter(f_auth_info); destination(authlog);
> destination(loghost); };
> log { source(src); filter(f_lpr_info); destination(lprlog);
> destination(loghost); };
> log { source(src); filter(f_news_info); destination(newslog);
> destination(loghost); };
> log { source(src); filter(f_cron_info); destination(cronlog);
> destination(loghost); };
>=20
> log { source(src); filter(f_local0); destination(local0log);
> destination(loghost); };
> log { source(src); filter(f_local2); destination(local2log);
> destination(loghost); };
> log { source(src); filter(f_local5); destination(local5log);
> destination(loghost); };
> log { source(src); filter(f_local6); destination(local6log);
> destination(loghost); };
> log { source(src); filter(f_local7); destination(local7log);
> destination(loghost); };
> log { source(src); filter(f_err); destination(alertlog);
destination(loghost);
> };
>=20
> log { source(src); destination(fallbacklog); flags(fallback); };
>=20
>=20
> log { source(net); filter(f_messages); destination(messages); };
> #log { source(net); filter(f_emerg); destination(console); };
> log { source(net); filter(f_kern_info); destination(kernlog);
> destination(loghost); };
> log { source(net); filter(f_user_info); destination(userlog);
> destination(loghost); };
> log { source(net); filter(f_mail_info); destination(maillog);
> destination(loghost); };
> log { source(net); filter(f_daemon_info); destination(daemonlog);
> destination(loghost); };
> log { source(net); filter(f_auth_info); destination(authlog);
> destination(loghost); };
> log { source(net); filter(f_lpr_info); destination(lprlog);
> destination(loghost); };
> log { source(net); filter(f_news_info); destination(newslog);
> destination(loghost); };
>=20
> log { source(net); filter(f_local0); destination(local0log);
> destination(loghost); };
> log { source(net); filter(f_local2); destination(local2log);
> destination(loghost); };
> log { source(net); filter(f_local5); destination(local5log);
> destination(loghost); };
> log { source(net); filter(f_local6); destination(local6log);
> destination(loghost); };
> log { source(net); filter(f_local7); destination(local7log);
> destination(loghost); };
> log { source(net); filter(f_err); destination(alertlog);
destination(loghost);
> };
>=20
> log { source(net); destination(fallbacklog); flags(fallback); };
>=20
> ## MYSQL
> #log { source(src); destination(d_mysql); };
> #log { source(net); destination(d_mysql); };
>=20
>=20
>=20
>=20
> _______________________________________________
> syslog-ng maillist - syslog-ng@lists.balabit.hu
> https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
>=20
--=20
Bazsi
_______________________________________________
syslog-ng maillist - syslog-ng@lists.balabit.hu
https://lists.balabit.hu/mailman/listinfo/syslog-ng
Frequently asked questions at http://www.campin.net/syslog-ng/faq.html