- syslog-ng - lists.balabit.hu

syslog-ng performance problem
by Santiago Gómez Cano 13 Feb '01

13 Feb '01

Hi I'm using the syslog-ng 1.40rc3-3 deb package running on a Debian GNU/Linux box collectin logs from 3 different web serves. When starting the syslgog-ng daemon it begins using a reasonable CPU amount (about 2 or 4%), sleeping most of the time and running when someone access our web pages and the logging server receives logs from the web servers. The CPU usage increases to 5 or 6% after one hour running and keeps on growing up to saturate the server with a 99.99% CPU usage about two or three ours after the start. I think the server has enough CPU and memory resources (Pentium III 650 with 1Gb RAM) to handle the logs incoming from the web servers as it works fine at the beginning without loosing messages and the performace decreases progressively instead of from the beginning. I think I could solve the problem with the appropiate gc_idle_threshold() and gc_busy_threshold() settings (now I using the default values), but I'm not sure if it would be the solution and in this case I don't know how to calculate the right values for these para meters. Any advice? -- Santiago Gómez Cano Systems Administrator / Administrador de Sistemas santiago.gomez(a)tiscali.es Tiscali Telecomunicaciones, S.A. www.tiscali.es T. +34 918370546 T. +34 699459573

1 0

strange logfile appearance
by Wiktor Wodecki 06 Feb '01

06 Feb '01

ello, I'm in the test-phase of my loggingserver. and syslog-ng started to create a logfile I don't understand why it did that. This is my conf, it's a fairly easy setup. (striped other unimportant stuff) options { long_hostnames(yes); sync(0); time_reap(10); use_time_recvd(yes); use_dns(yes); use_fqdn(yes); create_dirs(yes); log_fifo_size(10000); chain_hostnames(yes); }; source oldserver { udp(ip(xxx.xxx.xxx.xxx) port(514)); }; destination blackhole { file("/var/log/frontier/$YEAR$MONTH/$HOST"); }; destination blackhole_auth { file("/var/log/frontier/$YEAR$MONTH/$HOST-auth"); }; destination blackhole_crit { file("/var/log/frontier/$YEAR$MONTH/$HOST-crit"); }; destination blackhole_emerg { file("/var/log/frontier/$YEAR$MONTH/$HOST-emerg"); }; destination blackhole_err { file("/var/log/frontier/$YEAR$MONTH/$HOST-err"); }; destination blackhole_messages { file("/var/log/frontier/$YEAR$MONTH/$HOST-messages"); }; destination blackhole_warn { file("/var/log/frontier/$YEAR$MONTH/$HOST-warn"); } log { source(oldserver); destination(blackhole); }; log { source(oldserver); filter(f_auth); destination(blackhole_auth); }; log { source(oldserver); filter(f_crit); destination(blackhole_crit); }; log { source(oldserver); filter(f_emerg); destination(blackhole_emerg); }; log { source(oldserver); filter(f_err); destination(blackhole_err); }; log { source(oldserver); filter(f_messages); destination(blackhole_messages); }; log { source(oldserver); filter(f_warn); destination(blackhole_warn); }; The syslog-ng ran for the last 10 days without problems, however today I discovered three new logfiles. -rw------- 1 root root 119 Feb 1 10:43 last -rw------- 1 root root 60 Feb 1 05:45 last-messages -rw------- 1 root root 60 Feb 1 05:45 last-warn erwin:/var/log/frontier/200102# cat last Feb 1 05:45:23 last/ns2.xxx.net message repeated 2 times Feb 1 10:43:27 last/ns2.xxx.net message repeated 1 time erwin:/var/log/frontier/200102# cat last-messages Feb 1 05:45:23 last/ns2.xxx.net message repeated 2 times erwin:/var/log/frontier/200102# cat last-warn Feb 1 05:45:23 last/ns2.xxx.net message repeated 2 times The box the logs came from is a sparc10 with solaris5.8. The orginal logfile statements were: Feb 1 05:45:23 sparky last message repeated 2 times ... Feb 1 10:47:50 sparky last message repeated 1 time There are plenty of other "last message repeated..." lines in sparky's logfile, so I don't see a reason *why* those both got logged seperately. The first logentry repeated a warn-message, the other one a notice-message. Can anyone give me a hint please? The version I use is 1.4.10 on linux. (syslog-ng doesn't run yet on the sparc10. I export the logs via the goold old "*.crit,*.warn,etc,etc. @hostname way") -- Regards, Wiktor Wodecki, Unix Administration | Wapme-Systems AG Tel.: +49-211-748450 | Muensterstrasse 248 Fax: +49-211-74845176 | 40470 Duesseldorf E-Mail: wodecki(a)wapme-systems.de | http://www.wapme-systems.de

1 1

Core
by Brian E. Seppanen 01 Feb '01

01 Feb '01

Syslog-ng dumped on the 29th, and running gdb --core=core gives the following. What's signal 6? Any ideas appreciated. Core was generated by `syslog-ng -f /usr/local/etc/syslog-ng.conf -p /var/run/syslog-ng.pid'. Program terminated with signal 6, Aborted. #0 0x40068dd1 in ?? () This is on a redhat 7.0 box. Syslog-ng 1.4.10. Thanks... Brian Seppanen Charter Communications Regional Data Center 906-228-4226 ext 23 Marquette, MI seppy(a)chartermi.net

2 1

Re: bug? OR dont seem to work right.
by Samuel Krempp 29 Jan '01

29 Jan '01

You could not reproduce it ? maybe it is specific to my platform then.. PC Debian Woody (up to date), syslog-ng deb package version 1.4.10-1.. Here is more details on my setup and results, and I attach my complete syslog-ng.conf (only the begining of my syslog-ng.conf differs from debian's default config), and a full piece of 'try.log'. ------------------- filter f_tunnel { program("tunnel"); }; filter f_dhcp { program("dhcp"); }; filter f_squid { program("squid"); }; filter f_ssh { facility(local3); }; filter f_separe { program("(tunnel)|(dhcp)|(squid)"); }; filter f_try { filter(f_separe) or filter(f_ssh); }; log { source(src); filter(f_ssh); destination(sshlog); }; log { source(src); filter(f_separe); destination(separelog); }; log { source(src); filter(f_try); destination(trylog); }; ------------------- ----------- ssh.log is ok (full of sshd messages...) ----- separe.log : allright (dhcp, squid and stunnel messages only.) Jan 19 16:13:41 zamok dhcpd-2.2.x: DHCPACK on 138.231.138.32 to 52:54:05:f5:26:39 via eth0Jan 19 16:14:17 zamok stunnel[6495]: Connection closed: 191 bytes sent to SSL, 39 bytes sent to socket Jan 19 16:14:21 zamok stunnel[6500]: tunnel-pop connected from 212.232.35.241:3636 Jan 19 16:14:26 zamok stunnel[6500]: Connection closed: 938 bytes sent to SSL, 69 bytes sent to socket Jan 19 16:14:32 zamok stunnel[6514]: tunnel-pop connected from 138.231.137.11:1663 ----- try.log : how come ? (chosen bits : ) Jan 19 16:13:33 zamok syslog-ng[6453]: syslog-ng version 1.4.10 starting Jan 19 16:13:33 zamok imapd[6455]: connect from kino.crans.org Jan 19 16:13:35 zamok imapd[6458]: Logout user=szakacs host=localhost.crans.org [127.0.0.1] Jan 19 16:13:37 zamok sshd2[386]: connection from "138.231.137.199" Jan 19 16:13:37 zamok sshd[6460]: log: Generating 768 bit RSA key. Jan 19 16:14:17 zamok cucipop[6496]: arlais 127.0.0.1 0, 0 (0), 0 (0) Jan 19 16:14:17 zamok stunnel[6495]: Connection closed: 191 bytes sent to SSL, 39 bytes sent to socket Can you see where the problem comes from ? that would be very helpful for me to be able to use 'or' correctly.. -- Sam

2 1

Re: iptables/netfilter and syslog-ng
by Samuel Krempp 27 Jan '01

27 Jan '01

For syslog-ng to receive kernel logs, you need to either add a source for /proc/kmsg, or (better) run klogd. (it's almost written in the documentation..) klogd is better because it translates the kernel logs using System.map. The trick, on my debian, is that installing the package syslog-ng removes sysklogd, which contained both the old syslog and klogd, so I simply copied by hand the klogd daemon in /usr/local/sbin.. hth -- Sam

2 1

combining filters : completely broken ?
by Samuel Krempp 27 Jan '01

27 Jan '01

Okay, I've already posted a message about the bad behaviour of ORing filters on my debian (and got no response, perhaps I should have restrained pine from sending attachments vith MIME, but well.) I realise now that combining filters like this : destination newsnotice { file("/var/log/news/news.notice" owner("news") group("adm") perm(0640)); }; filter f_news { facility(news); }; filter f_notice { level(notice); }; log { source(src); filter(f_news); filter(f_notice); destination(newsnotice); }; will also break syslog (news.notice, suddenly, becomes very quite.. not growing at all.. strangely I suspect syslog-ng) I think combining filters wihtin syslog is broken, at least on my plaform, but not only. In a previous message, by michael.senn(a)cmg.nl, I read this : """ I have done some very basic testing with the not filter and suggest that you try spliting the not into it's own filter then combine them in the log statement as follows. It works for me with matching in an error message. """ Now, is there any good reason to accept this specificity about the 'not' from syslog-ng ? isnt this rather a sign that syslog-ng is broken when it comes to combining filters ?? I liked syslog-ng very well, but if filters cant be combined properly, it's much less useful.. aren't those a trivial thing to fix ?? or maybe syslog-ng depends on something for combining filters, and this 'something' is not really the same across different distribs ?? -- Samuel

1 1

iptables/netfilter and syslog-ng
by Laurence J. Lane 27 Jan '01

27 Jan '01

I'm having trouble with iptables (Linux kernel 2.4.x) logging with syslog-ng. sysklogd is logging as expected. As far as I can see, a syslog-ng config like this is logging everything except iptables: source src { unix-stream("/dev/log"); internal(); }; destination testing { file("/root/testing"); }; log { source(src); destination(testing); }; I've tried syslog levels 0-7 with iptables, but syslog-ng still doesn't show the traffic. Anyone have an idea of I'm doing wrong? -- ljlane(a)nontoxic.org

1 1

RE: [syslog-ng]Filter Problem
by Michael Senn 26 Jan '01

26 Jan '01

I am guess from your email that all of the or's in the filter work correctly and it is just the 'not facility(local7)' that is the probel area. I have done some very basic testing with the not filter and suggest that you try spliting the not into it's own filter then combine them in the log statement as follows. It works for me with matching in an error message. >My intent with the following filter, etc. is to exclude messages coming in >from facility local7 from being logged to /var/adm/messages. It does not >seem to work as those messages are making it there. Can anyone spot what >might be wrong with the filter/destination/log stanza(s)? >filter f_var_adm { level(emerg,alert,crit,err) > or ( facility(kern) and level(debug) ) > or ( facility(daemon) and level(notice) ) > or ( facility(local2) and level(notice) ) > or ( facility(auth) and level(notice,info) ) > or ( facility(mail) and level(crit) ) ; }; filter not_local7 { not facility(local7) ; }; >destination d_var_adm { file("/var/adm/messages"); }; ># Log all messages conforming to the f_var_adm filter to /var/adm/messages >log { source(stdlog); filter(not_local7); > filter(f_var_adm); > destination(d_var_adm); > }; _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng

2 1

RE: [syslog-ng]Filter Problem
by John A. Parker 26 Jan '01

26 Jan '01

> > My intent with the following filter, etc. is to exclude > > messages coming in > > from facility local7 from being logged to /var/adm/messages. > > It does not > > seem to work as those messages are making it there. Can > > anyone spot what > > might be wrong with the filter/destination/log stanza(s)? > > > > filter f_var_adm { level(emerg,alert,crit,err) > > or ( facility(kern) and level(debug) ) > > or ( facility(daemon) and level(notice) ) > > or ( facility(local2) and level(notice) ) > > or ( facility(auth) and level(notice,info) ) > > or ( facility(mail) and level(crit) ) > > and not ( facility(local7) ); }; > >are any of the messages arriving from local7 in emerg, alert, crit, or err >level? Yes.

1 0

RE: [syslog-ng]Filter Problem
by Andrew Fort 26 Jan '01

26 Jan '01

> My intent with the following filter, etc. is to exclude > messages coming in > from facility local7 from being logged to /var/adm/messages. > It does not > seem to work as those messages are making it there. Can > anyone spot what > might be wrong with the filter/destination/log stanza(s)? > > filter f_var_adm { level(emerg,alert,crit,err) > or ( facility(kern) and level(debug) ) > or ( facility(daemon) and level(notice) ) > or ( facility(local2) and level(notice) ) > or ( facility(auth) and level(notice,info) ) > or ( facility(mail) and level(crit) ) > and not ( facility(local7) ); }; are any of the messages arriving from local7 in emerg, alert, crit, or err level?

1 0