- syslog-ng - lists.balabit.hu

Fwd: new to list, and syslog-ng
by Brian Loe 02 Aug '06

02 Aug '06

<I hope you don't get this twice (hadn't confirmed my subscription the first time).> I'm hoping some of you can give me some guidance in my implementation of syslog-ng on an AIX 5.3 system. This is what I have for a config file thus far - will it work? Also, some help on the best way to move from the AIX syslogd to syslog-ng? I'm new to AIX as well... # # Global options. # options { long_hostnames(off); use_dns(no); chain_hostnames(no); sync(0); perm(0640); stats(3600); }; # # 'src' is our main source definition. you can add # more sources driver definitions to it, or define # your own sources, i.e.: # #source my_src { .... }; # source s_local { internal(); unix-stream("/dev/log"); }; source s_udp { udp(ip("0.0.0.0") port(514)); }; # # Filter definitions # filter f_cron { facility(cron); }; # filter f_local { facility(local0, local1, local2, local3, local4, local5, local6, local7) and not filter(f_cron); }; # filter f_messages { not facility(news, mail); }; filter f_warn { level(warn, err, crit); }; filter f_alert { level(alert); }; # # Cron-messages in one file: # destination cron { file("/var/log/cron"); }; log { source(s_lcoal); filter(f_cron); destination(cron); }; # # local[1-7]: # destination localmessages { file("/var/log/localmessages"); }; log { source(s_local); destination(localmessages); }; # # Network messages: # destination netmessages { file("/var/log/netmessages"); }; log { source(s_udp); destination(netmessages); }; # # Network message pipe: # destination net_pipe { file("/tmp/log/net_pipe"); }; log { source(s_udp); destination(net_pipe); };

1 0

syslog-ng 1.0rc1 stops listening to UDP when zero-length packet received
by Dark Angael 02 Aug '06

02 Aug '06

I have been testing 2.0 rc1 for use in a production environment and have discovered a potential DOS exploit. I discovered this when I stopped receiving logs after a particular application started. It turns out that for some reason this app was sending a zero-length packet to syslog. After receiving this packet, syslog-ng gives the following output in debug mode: EOF occurred while reading; fd='3' Closing log reader fd; fd='3' This is reproducible using packet injection also. It has been shown to happen on both solaris 8 and linux (2.6.15-gentoo-r5). It does not happen on syslog-ng version 1.6.11. _________________________________________________________________ New year, new job there's more than 100,00 jobs at SEEK http://a.ninemsn.com.au/b.aspx?URL=http%3A%2F%2Fninemsn%2Eseek%2Ecom%2Eau&_…

2 3

ISODATE, timezone offset and daylight saving
by Per Jessen 31 Jul '06

31 Jul '06

My local timezone is CET and as we've currently got daylights saving on, we're two hours ahead of UTC. So the offset - when using the ISODATE format - should be +0200. Instead it is +0100. I'm on glibc 2.4. Distro is SUSE 10.1, syslog-ng is 1.6.8. The machines hardware clock is in UTC, localtime time CEST. tournesol:/var/log # date Fri Jul 28 18:19:56 CEST 2006 tournesol:/var/log # date -u Fri Jul 28 16:20:01 UTC 2006 tournesol:/var/log # tournesol:/var/log # date +%z +0200 Am I likely to be doing something wrong? From /var/log/mail: 2006-07-28T18:06:39+0100 tournesol postfix/qmgr[7815]: 3E62737129: removed /Per Jessen, Zürich

2 2

Setting UDP Receive Buffer size, then when increase performance problems
by James 31 Jul '06

31 Jul '06

SUMMARY: Due to the large number of UDP packets (aprox: 6,700 per second) coming into our syslog-ng (version 1.6.4) running on an HP-UX 11.11 server we have had a desire to increase the receive buffer for UDP. I could NOT find a way to do this in syslog-ng, so we changed the default at the OS level (socket_udp_rcvbuf_default). Has it been considered (or possibly already added) to allow this option to be specified in the config file? Increasing this buffer size, seemed to cause a performance problem because syslog-ng (again at least on HP-UX 11.11) has POLLIN set to POLLRDNORM|POLLRDBAND (default when _XOPEN_SOURCE_EXTENDED is set) which must mean the OS walks the entire buffer looking for priority message when poll() is called. Since syslog-ng appears to do a poll(), then 1 read(), followed by another poll(), the OS is walking the entire buffer, and then only reading 1 entry. During the time the OS spends walking this buffer, hundreds of packets have been added. Thus, syslog-ng can reach a point of no return as long as data keeps coming in at this rate. I did a quick change in io.c to override the POLLIN setting HP-UX sets for syslog-ng to the value it would use if _XOPEN_SOURCE_EXTENDED was NOT set (0x0001). This change seems to work for us, but more details and maybe another option is mentioned in my details below. DETAILS: We have an HP-UX system that is regularly receiving 6,700 log entries per second from several systems via UDP. Originally we tried handling this workload with one daemon, but was CPU constrained due to it being single threaded. We then split the work among 3 daemons each on their own virtual IP so we could utilize the other CPUs in the system. We also increased socket_udp_rcvbuf_default (default receive buffer size for UDP) from 65K to 75mb - 150mb. It would be nice if the syslog-ng config file allowed us to set this value instead of impacting ALL UDP listeners on the system. I have edited our copy of io.c to include something like the following. This example is crude, but seems to work for us setting 75Mb buffer (I hard coded the value, but obtaining from the config file would be a better solution): int Buffer = 78643199; socklen_t Len = sizeof(Buffer); if ( (setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &Buffer, Len)) == 0 ) { Buffer=0; getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &Buffer, &Len); debug("io.c: SO_RCVBUF now set to: %i\n", Buffer); } else { werror("io.c: setsockopt() failed setting SO_RCVBUF (errno %i), %z\n", errno, strerror(errno)); } However, this can lead to another problem with syslog-ng (at least on on HP-UX). Once the buffer starts to fill the OS starts spending a LOT of CPU in system time. This appears to be because of the socket options specified due to _INCLUDE_XOPEN_SOURCE_EXTENDED being set during compile. On HP-UX 11.11 poll.h sets POLLIN to (POLLRDNORM|POLLRDBAND) when _INCLUDE_XOPEN_SOURCE_EXTENDED is defined. It is my understanding that this means the OS will walk the entire buffer looking for a priority message when syslog-ng calls poll() to look for the next packet. Once we increase the buffer past the 65K default in HP-UX this walking can take some time and lots of system CPU time (hate to see this if buffer was set the the 2GB maximum). Overriding the POLLIN options to the setting poll.h would use if _INCLUDE_XOPEN_SOURCE_EXTENDED was NOT defined (0x0001) seems to resolve this problem, and actually allows syslog-ng to empty out this large UDP buffer space very quickly. Another solution (which looks like may have been talked about in a post from John Morrissey) may be to change the way syslog-ng loops on the UDP socket. It appears to me that syslog-ng is currently doing: poll() read() Back to poll() This delay maybe could only be encountered once if syslog-ng was to do something like: poll() read() with O_NONBLOCK set if return -1 with EAGAIN, Back to poll() else Back to read() with O_NONBLOCK set I am not sure what impact this would have if syslog-ng was receiving data from more than one UDP socket (which is what we do) as it may spend too much time doing emptying the one buffer while the other data listeners wait. Have concerns like this already been addressed in more current releases? Do these issues I have seen HP-UX concerns for other operating systems, or should I just continue to modify the standard syslog-ng code as there would be little to no benefit for others? Thanks, /James

2 1

solaris package
by Joop Boonen 28 Jul '06

28 Jul '06

Hello All, I've just build the solaris package via solbuild/rules. When i was doing this i was wondering about a few things. These is a nice init.d file in the solbuild directory. solbuild/syslog-ng.init.d After the build it cant be found in the package at all. When i copy the above mentioned init.d script it uses the config file /opt/syslog-ng/etc/syslog-ng/syslog-ng.conf. Would /etc/syslog-ng/syslog-ng.com a better location? I adapted the script for this now. Why is the package in stalled in /opt/syslog-ng in stead of /usr/local? By the way Sun uses /opt/sfw for gnu software. Why are to init.d packages available in the source, being solbuild/syslog-ng.init.d and contrib/init.d.solaris . The first one is realted to /opt/syslog-ng the second one to /usr/local . Regards, Joop Boonen.

2 3

syslog hangup
by Vincent Régnard 28 Jul '06

28 Jul '06

Hi, Since at least a couple of month now, we've been experiencing some strange troubles on some of our routers which we now associate to a syslog-ng (or a syslog-ng internal facility) hangup. The version of syslog ng we use is 1.6.8 on different linux kernel 2.4.31 and 2.6.8. I read some posts on this list (https://lists.balabit.hu/pipermail/syslog-ng/2006-May/008784.html) establishing some similar problems. The conditions in which the problem occur are not clear to us. There are obviously different sircumstances that lead syslog-ng to hangup. We first noticed the problem on a server where some users where not able to login, certainly because of the impossibility to write to the /dev/log socket. Reloading syslog-ng when we had an active shell on the server corrected the problem. For this server we "solved" the problem by adding a cron to check the syslog activity and reload syslog if needed. This is not a nice soltution, but it avoids hard reboot of the server. More recently we realized that some routers where not logging some (iptables firewall) events sent to syslog, when in the same time, log from other daemons where treated correctly, again reloading syslog-ng fixes the problem untill the problem randmly accurs again. I am presently studdying the way these log messages are sent to syslog to understand this trouble better. We are tracking the causes of such an annoying behaviour without succes untill now. First of all we would like to understand what is happening in syslog-ng itselfs, at what level is this hangup ? kernel ? syslog ? is it related to /dev/log socket ? Maybe some experts or syslog devloppers can send us some hints ? Is it related to the kernel environement ? /proc ? udev ? Or is it possible that another daemon is responsible for this syslog hangup. Apparently the problem is also present in newer releases in the 1.6.X branch according to the posts on the list, I checked the branch changelogs without seeing anything on that. Has some work been devoted to fix this kind of trouble in more recent branches (1.9 and 2.0) ? We are planning to develop a daemon to monitor syslog-ng and reload the service in case of hangup. If some of you already performed some work in that direction, we would be glad to share the effort or learn the best and more efficient way proceed. Any hints, comments or suggestions are welcome. Thanks in advance. -- Vincent Régnard vregnard(a)tbs-internet.com TBS-internet.com 027 630 5902

2 1

flow-control and pipes
by Steen Thomsen 27 Jul '06

27 Jul '06

Hello I am currently exploring the possibilitys with flowcontrol from syslog-ng 2.0 on a freeBSD system. I have logs coming in from different sources to a centralized logserver, where I want to save the logs in an archive, and also read the logs into a database. What I seek to accomplish is that in case of database breakdown, logs should still be written to the archive, and when the database is up again, the missing logs should then automatically be written to the database, all in chronological order. The way I plan to do it, with the new possibilities flow-control offer, is that incoming logs are piped to a script that do 2 things. One is to keep count of the linenumbers in the archivefiles, and add to each log its own linenumber, before writting the log to its archivefile. The second task is to pipe a copy of the log (with the line number added) back to syslog-ng. Syslog-ng then pipes the log to another script that reads the linenumber from the log, and compare it to the linenumber from the previous log this script read. If the linenumbers are not consecutive, it means that some logs where lost on the way from the archivescript to this accounterscript. It therefore fetches the missing logs from the archive, identified by the linenumbers, before reading from the pipe again. If I use flow-control on the logs coming in from the sources and piped to the archive script, and no flow-control on the logs that are piped back to syslog-ng and further on to the accounterscript, I accomplish that logs are always written to the archive, even when the database fails, and all logs are later written to database when it comes up again, since there is account for all the logs through their linenumber. The basic idea being that everything is piped under normal conditions, and I only read from the archive in case of databaseproblems. The flow-control works fine for the first part of the process. I can have the archivescript write to the archive with no loss of logs. I can also let the archive script write the same log both to the archive and a pipe, when I place a script at the other end of the pipe to read the pipe, and still not loose logs.. But when I try to write to a pipe that syslog-ng shall read, the flowcontrol doesn't seem to work. Log are lost on the way to the archive script, even though its flowcontrolled. My question is, if there is something I have overlooked that might cause this behaviour. It works when I read the pipe from a script of my own, but not when syslog-ng reads it. The reason I want syslog-ng to read the pipe, and not simply a script of my own, is that syslog-ng will read from the pipe, even if the database is down, where my own script would block on the pipe, and consequently block for incoming logs, since that part ( the other end of the pipe where the archivescript is) is flow-controlled. I read somewhere that syslog-ng opens a pipe for both reading and writing, but I don't quite understand how this comply with the nature of pipes. Can anyone elaborate on this? And could that be a problem in my setup? Kind Regards Steen Thomsen softwaredeveloper <http://www.bridicum.com/> <http://www.bridicum.com/>

1 1

Need help pulling out message info before sending on to loghost from syslog-ng proxy
by Steven Meacham 27 Jul '06

27 Jul '06

Hi, Need help with removing some information before message is sent to the loghost server. I am trying to remove what looks like a priority or facility code date and time....is there any way to remove this out of the message before fowarding on to loghost Thanks Steven -- No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.394 / Virus Database: 268.10.4/396 - Release Date: 7/24/2006

2 1

Re: syslog-ng works only with one Core-CPU
by Rolf Heitfeld 27 Jul '06

27 Jul '06

>> Hi, >> >> we're using syslog-ng-1.6.2-29.3 (default on SUSE SLES9 Systems). >> >> We saw, that syslog-ng only uses one core of the Dual Core AMD Opteron(tm) Processor 275 of the System. >> >> Is there any possibility to turn this off? >> >> Thanx for any help! >> >> Rolf Heitfeld >Hello, >Can you clarify what is the problem? Syslog-ng isn't a multithreaded >program, so it runs only on one processor, as any singlethreaded program >does. Hello, Sorry perhaps I have expressed myself in a wrong way. The question for me was: Is there any Version of syslog-ng that is multithreaded and we could use instead? And if I unterstood you correctly the answer is "no". Thank you for answering! Regards, Rolf Heitfeld

2 1

RE: [syslog-ng] Hi Every Body - Host Name Rewrite Help!
by Nasir, Meraz 26 Jul '06

26 Jul '06

Hi I don't create any local file to capture logs, everything piped to mysql database so that I can query it using php-syslog-ng. ##################################### options { group(logs); dir_group(logs); perm(0640); dir_perm(0750); create_dirs(yes); use_time_recvd(yes); sync(0); use_fqdn(no); stats (7200); log_fifo_size(4096); keep_hostname(yes); chain_hostnames(no); bad_hostname("gconfd"); gc_idle_threshold(500); gc_busy_threshold(10000); }; ############################################################# Issue: I maintain all the device entry on the hosts file, usually syslog-ng rewrites hostname for the log source. For some reason it is not able to rewrite host names for the logs that is generated from all my APC devices. Thanks, Meraz 718.613.4661 -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alexander Clouter Sent: Wednesday, July 26, 2006 3:38 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Hi Every Body Hi, Sorry but this is not the greatest of support requests.... Nasir, Meraz <MNasir(a)INTERFAITHMEDICAL.com> [20060726 15:22:31 -0400]: > > Hi > > Bellow is my Environment: > > * Host: Linux version 2.6.9-5.ELsmp (bhcompile(a)decompose.build.redhat.com) (gcc version 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)) #1 SMP Wed Jan 5 19:30:39 EST 2005 > usually not relevant > * mysql Ver 14.7 Distrib 4.1.7, for redhat-linux-gnu (i386) > * httpd-2.2.0 > * php-5.1.2 > Not syslog's fault, unless you plan on dumping all your logs to an SQL database, but really it could be any database in practise. > * syslog-ng-1.6.9.orig > Useful bit. > My Issue: > > I am not able to re-write UPS's name, > to me an "UPS's" is a uninterruptable power supply....is that what you mean. As you don't put it in any context (why don't you get it to advertise a different name?) its hard to tell what you are doing. Usually when confronted by a support request with no detail I ask the four magic questions that seem applicable to *any* fault: 1. what is it you are actually trying to do? 2. how are you trying to go about doing it? 3. what are you expecting to happen? 4. what is actually happening? We all need details, examples, configuration extracts, errors being quoted. Whats going into syslog-ng and whats coming out? Explain why your "UPS's" are fixed with their name. We all would love to help but, I at least, don't have the foggiest of an idea in regards to what you are talking about. What little I can extract I think what you could use is the template() system and strap it to some rDNS lookups cached locally with a lightweight DNS server on the local machine. Of course I need to know what it is you are trying to do :( Cheers Alex > This message is for the named person's use only. It may contain > confidential, proprietary or legally privileged information. No > confidentiality or privilege is waived or lost by any mistransmission. > If you receive this message in error, please immediately delete it and > all copies of it from your system, destroy any hard copies of it and > notify the sender. You must not, directly or indirectly, use, disclose, > distribute, print, or copy any part of this message if you are not the > intended recipient. Interfaith Medical Center and any of its > subsidiaries each reserve the right to monitor all e-mail communications > through its networks. > This message is for the named person's use only. It may contain confidential, proprietary or legally privileged > information. o Confidentiality is waved or lost by any mistransmission. If you receive this > message in error, please immediately delete it and all copies of it from your system, destroy any hard copies of > it and notify the sender. You must not, directly or indirectly, use, disclose, distribute, orint, or copy any > part of this message if you are not the intended recipient. Interfaith Medical Center and any of its subsidiaries > each reserve the right to monitor all e-mail communications through its networks. > does this really need to be this long? Why can't people have "disclaimer at http://example.com/legal-mumble-jumble.html"? </rant> > _______________________________________________ > syslog-ng maillist - syslog-ng(a)lists.balabit.hu > https://lists.balabit.hu/mailman/listinfo/syslog-ng > Frequently asked questions at http://www.campin.net/syslog-ng/faq.html > _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0