July 2006 - syslog-ng - lists.balabit.hu

syslog hangup
by Vincent Régnard 28 Jul '06

28 Jul '06

Hi, Since at least a couple of month now, we've been experiencing some strange troubles on some of our routers which we now associate to a syslog-ng (or a syslog-ng internal facility) hangup. The version of syslog ng we use is 1.6.8 on different linux kernel 2.4.31 and 2.6.8. I read some posts on this list (https://lists.balabit.hu/pipermail/syslog-ng/2006-May/008784.html) establishing some similar problems. The conditions in which the problem occur are not clear to us. There are obviously different sircumstances that lead syslog-ng to hangup. We first noticed the problem on a server where some users where not able to login, certainly because of the impossibility to write to the /dev/log socket. Reloading syslog-ng when we had an active shell on the server corrected the problem. For this server we "solved" the problem by adding a cron to check the syslog activity and reload syslog if needed. This is not a nice soltution, but it avoids hard reboot of the server. More recently we realized that some routers where not logging some (iptables firewall) events sent to syslog, when in the same time, log from other daemons where treated correctly, again reloading syslog-ng fixes the problem untill the problem randmly accurs again. I am presently studdying the way these log messages are sent to syslog to understand this trouble better. We are tracking the causes of such an annoying behaviour without succes untill now. First of all we would like to understand what is happening in syslog-ng itselfs, at what level is this hangup ? kernel ? syslog ? is it related to /dev/log socket ? Maybe some experts or syslog devloppers can send us some hints ? Is it related to the kernel environement ? /proc ? udev ? Or is it possible that another daemon is responsible for this syslog hangup. Apparently the problem is also present in newer releases in the 1.6.X branch according to the posts on the list, I checked the branch changelogs without seeing anything on that. Has some work been devoted to fix this kind of trouble in more recent branches (1.9 and 2.0) ? We are planning to develop a daemon to monitor syslog-ng and reload the service in case of hangup. If some of you already performed some work in that direction, we would be glad to share the effort or learn the best and more efficient way proceed. Any hints, comments or suggestions are welcome. Thanks in advance. -- Vincent Régnard vregnard(a)tbs-internet.com TBS-internet.com 027 630 5902

2 1

flow-control and pipes
by Steen Thomsen 27 Jul '06

27 Jul '06

Hello I am currently exploring the possibilitys with flowcontrol from syslog-ng 2.0 on a freeBSD system. I have logs coming in from different sources to a centralized logserver, where I want to save the logs in an archive, and also read the logs into a database. What I seek to accomplish is that in case of database breakdown, logs should still be written to the archive, and when the database is up again, the missing logs should then automatically be written to the database, all in chronological order. The way I plan to do it, with the new possibilities flow-control offer, is that incoming logs are piped to a script that do 2 things. One is to keep count of the linenumbers in the archivefiles, and add to each log its own linenumber, before writting the log to its archivefile. The second task is to pipe a copy of the log (with the line number added) back to syslog-ng. Syslog-ng then pipes the log to another script that reads the linenumber from the log, and compare it to the linenumber from the previous log this script read. If the linenumbers are not consecutive, it means that some logs where lost on the way from the archivescript to this accounterscript. It therefore fetches the missing logs from the archive, identified by the linenumbers, before reading from the pipe again. If I use flow-control on the logs coming in from the sources and piped to the archive script, and no flow-control on the logs that are piped back to syslog-ng and further on to the accounterscript, I accomplish that logs are always written to the archive, even when the database fails, and all logs are later written to database when it comes up again, since there is account for all the logs through their linenumber. The basic idea being that everything is piped under normal conditions, and I only read from the archive in case of databaseproblems. The flow-control works fine for the first part of the process. I can have the archivescript write to the archive with no loss of logs. I can also let the archive script write the same log both to the archive and a pipe, when I place a script at the other end of the pipe to read the pipe, and still not loose logs.. But when I try to write to a pipe that syslog-ng shall read, the flowcontrol doesn't seem to work. Log are lost on the way to the archive script, even though its flowcontrolled. My question is, if there is something I have overlooked that might cause this behaviour. It works when I read the pipe from a script of my own, but not when syslog-ng reads it. The reason I want syslog-ng to read the pipe, and not simply a script of my own, is that syslog-ng will read from the pipe, even if the database is down, where my own script would block on the pipe, and consequently block for incoming logs, since that part ( the other end of the pipe where the archivescript is) is flow-controlled. I read somewhere that syslog-ng opens a pipe for both reading and writing, but I don't quite understand how this comply with the nature of pipes. Can anyone elaborate on this? And could that be a problem in my setup? Kind Regards Steen Thomsen softwaredeveloper <http://www.bridicum.com/> <http://www.bridicum.com/>

1 1

Need help pulling out message info before sending on to loghost from syslog-ng proxy
by Steven Meacham 27 Jul '06

27 Jul '06

Hi, Need help with removing some information before message is sent to the loghost server. I am trying to remove what looks like a priority or facility code date and time....is there any way to remove this out of the message before fowarding on to loghost Thanks Steven -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.394 / Virus Database: 268.10.4/396 - Release Date: 7/24/2006

2 1

Re: syslog-ng works only with one Core-CPU
by Rolf Heitfeld 27 Jul '06

27 Jul '06

>> Hi, >> >> we're using syslog-ng-1.6.2-29.3 (default on SUSE SLES9 Systems). >> >> We saw, that syslog-ng only uses one core of the Dual Core AMD Opteron(tm) Processor 275 of the System. >> >> Is there any possibility to turn this off? >> >> Thanx for any help! >> >> Rolf Heitfeld >Hello, >Can you clarify what is the problem? Syslog-ng isn't a multithreaded >program, so it runs only on one processor, as any singlethreaded program >does. Hello, Sorry perhaps I have expressed myself in a wrong way. The question for me was: Is there any Version of syslog-ng that is multithreaded and we could use instead? And if I unterstood you correctly the answer is "no". Thank you for answering! Regards, Rolf Heitfeld

2 1

RE: [syslog-ng] Hi Every Body - Host Name Rewrite Help!
by Nasir, Meraz 26 Jul '06

26 Jul '06

Hi I don't create any local file to capture logs, everything piped to mysql database so that I can query it using php-syslog-ng. ##################################### options { group(logs); dir_group(logs); perm(0640); dir_perm(0750); create_dirs(yes); use_time_recvd(yes); sync(0); use_fqdn(no); stats (7200); log_fifo_size(4096); keep_hostname(yes); chain_hostnames(no); bad_hostname("gconfd"); gc_idle_threshold(500); gc_busy_threshold(10000); }; ############################################################# Issue: I maintain all the device entry on the hosts file, usually syslog-ng rewrites hostname for the log source. For some reason it is not able to rewrite host names for the logs that is generated from all my APC devices. Thanks, Meraz 718.613.4661 -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alexander Clouter Sent: Wednesday, July 26, 2006 3:38 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Hi Every Body Hi, Sorry but this is not the greatest of support requests.... Nasir, Meraz <MNasir(a)INTERFAITHMEDICAL.com> [20060726 15:22:31 -0400]: > > Hi > > Bellow is my Environment: > > * Host: Linux version 2.6.9-5.ELsmp (bhcompile(a)decompose.build.redhat.com) (gcc version 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)) #1 SMP Wed Jan 5 19:30:39 EST 2005 > usually not relevant > * mysql Ver 14.7 Distrib 4.1.7, for redhat-linux-gnu (i386) > * httpd-2.2.0 > * php-5.1.2 > Not syslog's fault, unless you plan on dumping all your logs to an SQL database, but really it could be any database in practise. > * syslog-ng-1.6.9.orig > Useful bit. > My Issue: > > I am not able to re-write UPS's name, > to me an "UPS's" is a uninterruptable power supply....is that what you mean. As you don't put it in any context (why don't you get it to advertise a different name?) its hard to tell what you are doing. Usually when confronted by a support request with no detail I ask the four magic questions that seem applicable to *any* fault: 1. what is it you are actually trying to do? 2. how are you trying to go about doing it? 3. what are you expecting to happen? 4. what is actually happening? We all need details, examples, configuration extracts, errors being quoted. Whats going into syslog-ng and whats coming out? Explain why your "UPS's" are fixed with their name. We all would love to help but, I at least, don't have the foggiest of an idea in regards to what you are talking about. What little I can extract I think what you could use is the template() system and strap it to some rDNS lookups cached locally with a lightweight DNS server on the local machine. Of course I need to know what it is you are trying to do :( Cheers Alex > This message is for the named person's use only. It may contain > confidential, proprietary or legally privileged information. No > confidentiality or privilege is waived or lost by any mistransmission. > If you receive this message in error, please immediately delete it and > all copies of it from your system, destroy any hard copies of it and > notify the sender. You must not, directly or indirectly, use, disclose, > distribute, print, or copy any part of this message if you are not the > intended recipient. Interfaith Medical Center and any of its > subsidiaries each reserve the right to monitor all e-mail communications > through its networks. > This message is for the named person's use only. It may contain confidential, proprietary or legally privileged > information. o Confidentiality is waved or lost by any mistransmission. If you receive this > message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of > it and notify the sender. You must not, directly or indirectly, use, disclose, distribute, orint, or copy any > part of this message if you are not the intended recipient. Interfaith Medical Center and any of its subsidiaries > each reserve the right to monitor all e-mail communications through its networks. > does this really need to be this long? Why can't people have "disclaimer at http://example.com/legal-mumble-jumble.html"? </rant> > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

RE: [syslog-ng] Hi Every Body
by Nasir, Meraz 26 Jul '06

26 Jul '06

Hi Bellow is my Environment: * Host: Linux version 2.6.9-5.ELsmp (bhcompile(a)decompose.build.redhat.com) (gcc version 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)) #1 SMP Wed Jan 5 19:30:39 EST 2005 * mysql Ver 14.7 Distrib 4.1.7, for redhat-linux-gnu (i386) * httpd-2.2.0 * php-5.1.2 * syslog-ng-1.6.9.orig My Issue: I am not able to re-write UPS's name, Thanks, Meraz 718.613.4661 ________________________________ From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Nasir, Meraz Sent: Wednesday, July 26, 2006 3:06 PM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] Hi Every Body Importance: High Hello I need some help on the configuration. I am not sure about the process Regards, Meraz Nasir Manager of Systems & Network Information Technology Services (ITS) Interfaith Medical Center 1545 Atlantic Avenue Brooklyn, NY 11213 718.613.4661 This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. No confidentiality or privilege is waived or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it and notify the sender. You must not, directly or indirectly, use, disclose, distribute, print, or copy any part of this message if you are not the intended recipient. Interfaith Medical Center and any of its subsidiaries each reserve the right to monitor all e-mail communications through its networks. This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. o Confidentiality is waved or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it and notify the sender. You must not, directly or indirectly, use, disclose, distribute, orint, or copy any part of this message if you are not the intended recipient. Interfaith Medical Center and any of its subsidiaries each reserve the right to monitor all e-mail communications through its networks.

2 1

Hi Every Body
by Nasir, Meraz 26 Jul '06

26 Jul '06

Hello I need some help on the configuration. I am not sure about the process Regards, Meraz Nasir Manager of Systems & Network Information Technology Services (ITS) Interfaith Medical Center 1545 Atlantic Avenue Brooklyn, NY 11213 718.613.4661 This message is for the named person's use only. It may contain confidential, proprietary or legally privileged information. o Confidentiality is waved or lost by any mistransmission. If you receive this message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of it and notify the sender. You must not, directly or indirectly, use, disclose, distribute, orint, or copy any part of this message if you are not the intended recipient. Interfaith Medical Center and any of its subsidiaries each reserve the right to monitor all e-mail communications through its networks.

1 0

Re: Re: [syslog-ng] syslog-ng 2.0rc1 memory usage
by Martin, David M 25 Jul '06

25 Jul '06

Configs included.. Similar behavior noticed on my RHEL4 instance. Both are running 2.0rc1 [Solaris 9 Host] PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND 29935 root 1 59 0 221M 220M sleep 16:49 2.38% syslog-ng.2.0rc # cat syslog-ng.conf options { sync(0); time_reopen(10); log_fifo_size(8192); flush_lines(4096); long_hostnames(off); use_dns(yes); use_fqdn(no); create_dirs(no); keep_hostname(yes); owner(syslogng); group(syslogng); }; source local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; source network { udp(); tcp(); }; # # standard destinations for local standard system messages # destination authlog { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog"); }; destination kern { file("/var/log/kern.log"); }; destination maillog { file("/var/log/maillog"); }; destination messages { file("/var/log/messages" perm(0644)); }; # # console destination # destination console { file("/dev/sysmsg"); }; # # filters for standard local system messages which come # in on non-local facilities # filter f_authpriv { facility(auth) ; }; filter f_syslog { not facility(auth) and not facility(mail); }; filter f_kern { facility(kern); }; filter f_mail { facility(mail); }; # # catch the rest # filter f_messages { level(info .. warn) and not facility(auth); }; # # filters for various emergency level messages # filter f_emergency { level(emerg); }; # # log emergency level messages out to console # log { source(local); filter(f_emergency); destination(console); }; # # log messages from local machine # log { source(local); filter(f_authpriv); destination(authlog); }; log { source(local); filter(f_syslog); destination(syslog); }; log { source(local); filter(f_kern); destination(kern); }; log { source(local); filter(f_mail); destination(maillog); }; # # catch the rest of the messages # log { source(local); source(network); filter(f_messages); destination(messages); flags(flow-control); }; destination hosts { file("/var/log/messages" owner(syslogng) group(syslogng) perm(0644) dir_perm(0700)); }; # # logs all incoming messages from network source to the sorted # destination # log { source(network); destination(hosts); }; destination d_sec { program( "/opt/sec/2.3.3/sec.pl -input=\"-\" -conf=/opt/sec/etc/sec.conf" template("$MONTH$DAY $UNIXTIME $HOST $MSG\n") ); }; filter f_mysql { host("XXXXXXXX") or host("XXXXXXXX"); }; log { source(network); filter(f_mysql); destination(d_sec); }; [RHEL 4 Update 3 Host] PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 16414 root 15 0 54064 50m 1068 S 1.3 1.2 3:51.60 syslog-ng # cat syslog-ng.conf ############################################################################### # options - global options { log_fifo_size(8192); create_dirs(no); #default chain_hostnames(no); #default yes normalize_hostnames(yes); #default requires 1.9.9 keep_hostname(yes); #default no owner(root); group(root); perm(0600); use_time_recvd(yes); #default no }; ############################################################################### # sources source abs_udp514 { udp(ip(0.0.0.0) port(514)); }; source abs_localhost { unix-stream("/dev/log"); internal(); }; ############################################################################### # filters filter abf_messages { level(debug) and not (facility(mail,auth,authpriv,cron,user)); }; filter abf_auth { facility(auth,authpriv); }; filter abf_mail { facility(mail); }; filter abf_cron { facility(cron); }; filter abf_spooler { facility(uucp) or (facility(news) and level(crit)); }; filter abf_boot { facility(local7); }; filter abf_daemon { facility(daemon); }; filter abf_user { facility(user); }; filter abf_ignore { not (host("2006")); }; filter abf_postfix { program ("postfix/"); }; ############################################################################### # destinations destination abd_messages { file( "/var/log/messages" group(bb) perm(0640) ); }; destination abd_auth { file("/var/log/auth.log"); }; destination abd_mail { file( "/var/log/mail.log" group(bb) perm(0640) ); }; destination abd_cron { file("/var/log/cron.log"); }; destination abd_spooler { file("/var/log/spooler.log"); }; destination abd_boot { file("/var/log/boot.log"); }; destination abd_daemon { file("/var/log/daemon.log"); }; destination abd_user { file("/var/log/user.log"); }; destination abd_sec { program( "/opt/sec/2.3.3/sec.pl -input=\"-\" -conf=/opt/sec/etc/sec.conf" template("$MONTH$DAY $UNIXTIME $HOST $MSG\n") ); }; destination abd_localhost { file("/var/log/localhost.log"); }; ############################################################################### # logs log { source(abs_udp514); filter(abf_messages); destination(abd_messages); }; log { source(abs_udp514); filter(abf_auth); destination(abd_auth); }; log { source(abs_udp514); filter(abf_mail); destination(abd_mail); }; log { source(abs_udp514); filter(abf_cron); destination(abd_cron); }; log { source(abs_udp514); filter(abf_spooler); destination(abd_spooler); }; log { source(abs_udp514); filter(abf_boot); destination(abd_boot); }; log { source(abs_udp514); filter(abf_daemon); destination(abd_daemon); }; log { source(abs_udp514); filter(abf_user); destination(abd_user); }; log { source(abs_udp514); filter(abf_postfix); destination(abd_sec); }; log { source(abs_localhost); destination(abd_localhost); }; ------------------------------ Message: 2 Date: Fri, 21 Jul 2006 11:16:27 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] syslog-ng 2.0rc1 memory usage To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1153473387.6682.6.camel(a)bzorp.balabit> Content-Type: text/plain On Thu, 2006-07-20 at 19:47 -0500, Martin, David M wrote: > Anyone running 2.0rc1 on Solaris 9 (sparc)? > If so, what memory requirements are you seeing for the syslog-ng > process? > > My install handles ~ 700 messages/sec (all UDP) and over the course of > a day syslog-ng will gradually consume nearly all available free > memory (I've seen it up to 1GB). > > (syslog-ng is bounced by logrotate daily, so the process repeats > itself) > I was able to substantially decrease the rate of consumption by adding > the 'flush_lines" global (syslog-ng process now at 91MB and growing > ... slowly). > This definitely seems to be some kind of memory leak. Can you post a configuration file that reproduces the problem? Hopefully it is a leak in syslog-ng itself and not in one of the system libraries. -- Bazsi ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 15, Issue 15 ***************************************** The information transmitted (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is intended only for the person(s) or entity/entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

2 1

syslog-ng works only with one Core-CPU
by Rolf Heitfeld 25 Jul '06

25 Jul '06

Hi, we're using syslog-ng-1.6.2-29.3 (default on SUSE SLES9 Systems). We saw, that syslog-ng only uses one core of the Dual Core AMD Opteron(tm) Processor 275 of the System. Is there any possibility to turn this off? Thanx for any help! Rolf Heitfeld

2 1

syslog-ng works only with one Core-CPU
by Rolf Heitfeld 25 Jul '06

25 Jul '06

Hi, we're using syslog-ng-1.6.2-29.3 (default on SUSE SLES9 Systems). We saw, that syslog-ng only uses one Core of the Dual Core AMD Opteron(tm) Processor 275 of the System. Is there any possibility to turn this off? Rolf Heitfeld LINUX System Engineer / OPS-Infrastructure www.combots.com <http://www.combots.com/> ComBOTS AG Amalienbadstraße 41 76227 Karlsruhe, Germany Tel. +49 721 5160 9935 Fax. +49 721 5160 Mobile + 49 173 385 02 45 Rolf.Heitfeld(a)combots.com <mailto:Rolf.Heitfeld@combots.com>

1 0