July 2006 - syslog-ng - lists.balabit.hu

Growing log files (i.e. access.log) and regex.
by Robert.Becker＠motoristsgroup.com 08 Mar '07

08 Mar '07

Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************

6 5

Analysis of logs and alerting
by Muath Al Khalaf 11 Feb '07

11 Feb '07

Hi, I am implementing a syslog-ng centralized correlation server. The server receives the log entries from some servers, apps and devices. Is there any good product (commercial or open-source) with GUI or web interface to analyze these logs and give real time alerts with strong data mining tools? I need a tool which could be used by both help desk non-technical staff and technical engineers. It should give basic alerting on applications' or devices' failures first for help desk, and then engineers could dig deeper into these alerts to understand the problem. Kind regards

4 3

Re: syslog-ng Digest, Vol 15, Issue 12
by dwarnock＠harbrook.net 28 Nov '06

28 Nov '06

I am on holiday from the 17th August returning on Monday the 31st August. If your query is urgent please contact the office on +44 (0)1423 508060 or Neil Harvey on +44 (0)15395 64734 (nharvey(a)harbrook.net) Regards David Warnock

5 9

ignore src{}; contents at startup?
by Brian A. Seklecki 25 Oct '06

25 Oct '06

All: On FreeBSD, /dev/klog provides for output from the kernel (normally redirected to /dev/console before syslog starts). At startup, /etc/rc.d/dmesg copies the kernel boot hardware paramters to /var/run/dmesg.boot. If I have my src(); set to: source src { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; ...at starup, syslog-ng drains the contents of /dev/klog. By default it tsends to them to user.notice. I like to log{} these to an SMS pager (so I know if a RAID is failing or other hardware problem). However, I'd like to avoid duplication of the efforts of /etc/rc.d/dmesg on behalf of syslog-ng(8) ...especially since I route user.notice to my pager, and kernel boot dmesg(8) can grow over 50 lines. /dev/klog really isn't a socket. It's a special char device. Is there anyway to tell syslog-ng(8) to ignore it's contents at startup? If it was tail(1), i'd say "tail -0 -f /file". Perhaps clear out the contents of /dev/klog somehow? TIA, ~BAS

2 2

couple install questions
by Nick Baronian 06 Sep '06

06 Sep '06

I have a RedHat Ent.4 box I want to put syslog-ng on. Should I remove syslogd (sysklogd) first? If so how do I get syslog-ng to handle the cron (vixie) and initscripts? How can I find out what configure options are available to me for syslog-ng? I figured I would just try compiling syslog-ng on the machine and see what happens with syslogd. I built glib-2.12.0 & eventlog-0.2.4 and then exported PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig I did a ./configure --enable-tcp-wrapper. But during make I get this error gcc -g -O2 -Wall -g -o syslog-ng main.o libsyslog-ng.a -lnsl -lrt -Wl,-Bstatic -lfl -L/usr/local/lib -lglib-2.0 -L/usr/local/lib -levtlog -Wl,-Bdynamic libsyslog-ng.a(logmsg.o)(.text+0x6a): In function `log_stamp_format': /root/syslog-ng-2.0rc1/src/logmsg.c:111: undefined reference to `g_assert_warning' libsyslog-ng.a(logmsg.o)(.text+0x2c2): In function `log_msg_ref': /root/syslog-ng-2.0rc1/src/logmsg.c:489: undefined reference to `g_assert_warning' libsyslog-ng.a(logmsg.o)(.text+0x35e): In function `log_msg_unref': /root/syslog-ng-2.0rc1/src/logmsg.c:503: undefined reference to `g_assert_warning' libsyslog-ng.a(logmsg.o)(.text+0xd9a): In function `log_msg_ack_block_end': /root/syslog-ng-2.0rc1/src/logmsg.c:623: undefined reference to `g_return_if_fail_warning' libsyslog-ng.a(logpipe.o)(.text+0x4f): In function `log_pipe_ref': /root/syslog-ng-2.0rc1/src/logpipe.c:38: undefined reference to `g_assert_warning' libsyslog-ng.a(logpipe.o)(.text+0x9b): In function `log_pipe_unref': /root/syslog-ng-2.0rc1/src/logpipe.c:47: undefined reference to `g_assert_warning' libsyslog-ng.a(center.o)(.text+0x7d): In function `log_endpoint_free': /root/syslog-ng-2.0rc1/src/center.c:90: undefined reference to `g_assert_warning' libsyslog-ng.a(center.o)(.text+0x175): In function `log_connection_new': /root/syslog-ng-2.0rc1/src/center.c:132: undefined reference to `g_assert_warning' libsyslog-ng.a(templates.o)(.text+0x2a6): In function `log_template_ref': /root/syslog-ng-2.0rc1/src/templates.c:169: undefined reference to `g_assert_warning' libsyslog-ng.a(templates.o)(.text+0x30e):/root/syslog-ng-2.0rc1/src/templates.c:180: more undefined references to `g_assert_warning' follow collect2: ld returned 1 exit status make[2]: *** [syslog-ng] Error 1 make[2]: Leaving directory `/root/syslog-ng-2.0rc1/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/root/syslog-ng-2.0rc1' make: *** [all] Error 2 Does anyone have any ideas on what is causing this? Thanks, Nick Baronian

4 3

$MSG parsing question
by Hubert Lang 08 Aug '06

08 Aug '06

Hello, i have a few questions about the message parser, basically i want to parse/split up the MESSAGE field itself and write the splitted up message in a mysql database, i cant find any documents about how this can be done, do i need an external parser (perl or whatever) or can this be done within syslog-ng.conf? right now i just can write the whole message to the mysqldb syslog-ng.conf template("INSERT INTO logs (host, facility, priority, level, tag, date,time, program, msg) VALUES ( '$HOST', '$FACILITY','$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); so it gets written to the database in this way: INSERT INTO logs (host, facility, priority, level, tag, date,time, program, msg) VALUES ( '10.44.10.253', 'local4','notice', 'notice', 'a5', '2006-06-29', '14:39:46', 'NS25', 'NS25: NetScreen device_id=NS25 [Root]system-notification-00257(traffic): start_time=\"2006-06-29 14:38:38\" duration=0 policy_id=95 service=http proto=6 src zone=Untrust dst zone=Untrust action=Deny sent=0 rcvd=0 src=10.10.10.225 dst=208.174.52.61 src_port=2042 dst_port=80 session_id=0' ); now i want to split up the message part itself system-notification traffic, insert the start_time/duration/policy_id/service/proto/src-zone etc etc in a different table this perl script http://www.optekconsulting.com/tools/nstf.pl has every field i need, Any Help is really welcome Cheers Hubert -- Die e-Mail-Boxes von Brennercom sind Virus-gesichert und Spam-gefiltert. Le caselle e-Mail di Brennercom sono protette da sistemi antivirus e antispam. http://www.brennercom.it

5 5

syslog-ng 2.0rc1 memory usage
by Martin, David M 04 Aug '06

04 Aug '06

Anyone running 2.0rc1 on Solaris 9 (sparc)? If so, what memory requirements are you seeing for the syslog-ng process? My install handles ~ 700 messages/sec (all UDP) and over the course of a day syslog-ng will gradually consume nearly all available free memory (I've seen it up to 1GB). (syslog-ng is bounced by logrotate daily, so the process repeats itself) I was able to substantially decrease the rate of consumption by adding the 'flush_lines" global (syslog-ng process now at 91MB and growing ... slowly). I installed 2.0rc1 on one server, kept another at 1.6.8 (roughly 500 messages/sec load). The syslog-ng process there is stable at 3 MB. Here's the config from the 2.0rc1 server: options { log_fifo_size(8192); flush_lines(4096); long_hostnames(off); use_dns(yes); use_fqdn(no); create_dirs(no); }; source network { udp(); }; filter f_messages { level(info..warn) and not facility(auth); }; destination messages { file("/var/log/messages"); }; log { source(network); filter(f_messages); destination(messages); }; syslog-ng 1.6.8 config identical except for options section: options { use_fqdn(no); keep_hostname(no); use_dns(yes); long_hostnames(off); sync(0); log_fifo_size(1000); }; The information transmitted (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is intended only for the person(s) or entity/entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

2 2

ISODATE, timezone offset and daylight saving
by Per Jessen 31 Jul '06

31 Jul '06

My local timezone is CET and as we've currently got daylights saving on, we're two hours ahead of UTC. So the offset - when using the ISODATE format - should be +0200. Instead it is +0100. I'm on glibc 2.4. Distro is SUSE 10.1, syslog-ng is 1.6.8. The machines hardware clock is in UTC, localtime time CEST. tournesol:/var/log # date Fri Jul 28 18:19:56 CEST 2006 tournesol:/var/log # date -u Fri Jul 28 16:20:01 UTC 2006 tournesol:/var/log # tournesol:/var/log # date +%z +0200 Am I likely to be doing something wrong? From /var/log/mail: 2006-07-28T18:06:39+0100 tournesol postfix/qmgr[7815]: 3E62737129: removed /Per Jessen, Zürich

2 2

Setting UDP Receive Buffer size, then when increase performance problems
by James 31 Jul '06

31 Jul '06

SUMMARY: Due to the large number of UDP packets (aprox: 6,700 per second) coming into our syslog-ng (version 1.6.4) running on an HP-UX 11.11 server we have had a desire to increase the receive buffer for UDP. I could NOT find a way to do this in syslog-ng, so we changed the default at the OS level (socket_udp_rcvbuf_default). Has it been considered (or possibly already added) to allow this option to be specified in the config file? Increasing this buffer size, seemed to cause a performance problem because syslog-ng (again at least on HP-UX 11.11) has POLLIN set to POLLRDNORM|POLLRDBAND (default when _XOPEN_SOURCE_EXTENDED is set) which must mean the OS walks the entire buffer looking for priority message when poll() is called. Since syslog-ng appears to do a poll(), then 1 read(), followed by another poll(), the OS is walking the entire buffer, and then only reading 1 entry. During the time the OS spends walking this buffer, hundreds of packets have been added. Thus, syslog-ng can reach a point of no return as long as data keeps coming in at this rate. I did a quick change in io.c to override the POLLIN setting HP-UX sets for syslog-ng to the value it would use if _XOPEN_SOURCE_EXTENDED was NOT set (0x0001). This change seems to work for us, but more details and maybe another option is mentioned in my details below. DETAILS: We have an HP-UX system that is regularly receiving 6,700 log entries per second from several systems via UDP. Originally we tried handling this workload with one daemon, but was CPU constrained due to it being single threaded. We then split the work among 3 daemons each on their own virtual IP so we could utilize the other CPUs in the system. We also increased socket_udp_rcvbuf_default (default receive buffer size for UDP) from 65K to 75mb - 150mb. It would be nice if the syslog-ng config file allowed us to set this value instead of impacting ALL UDP listeners on the system. I have edited our copy of io.c to include something like the following. This example is crude, but seems to work for us setting 75Mb buffer (I hard coded the value, but obtaining from the config file would be a better solution): int Buffer = 78643199; socklen_t Len = sizeof(Buffer); if ( (setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &Buffer, Len)) == 0 ) { Buffer=0; getsockopt(fd, SOL_SOCKET, SO_RCVBUF, &Buffer, &Len); debug("io.c: SO_RCVBUF now set to: %i\n", Buffer); } else { werror("io.c: setsockopt() failed setting SO_RCVBUF (errno %i), %z\n", errno, strerror(errno)); } However, this can lead to another problem with syslog-ng (at least on on HP-UX). Once the buffer starts to fill the OS starts spending a LOT of CPU in system time. This appears to be because of the socket options specified due to _INCLUDE_XOPEN_SOURCE_EXTENDED being set during compile. On HP-UX 11.11 poll.h sets POLLIN to (POLLRDNORM|POLLRDBAND) when _INCLUDE_XOPEN_SOURCE_EXTENDED is defined. It is my understanding that this means the OS will walk the entire buffer looking for a priority message when syslog-ng calls poll() to look for the next packet. Once we increase the buffer past the 65K default in HP-UX this walking can take some time and lots of system CPU time (hate to see this if buffer was set the the 2GB maximum). Overriding the POLLIN options to the setting poll.h would use if _INCLUDE_XOPEN_SOURCE_EXTENDED was NOT defined (0x0001) seems to resolve this problem, and actually allows syslog-ng to empty out this large UDP buffer space very quickly. Another solution (which looks like may have been talked about in a post from John Morrissey) may be to change the way syslog-ng loops on the UDP socket. It appears to me that syslog-ng is currently doing: poll() read() Back to poll() This delay maybe could only be encountered once if syslog-ng was to do something like: poll() read() with O_NONBLOCK set if return -1 with EAGAIN, Back to poll() else Back to read() with O_NONBLOCK set I am not sure what impact this would have if syslog-ng was receiving data from more than one UDP socket (which is what we do) as it may spend too much time doing emptying the one buffer while the other data listeners wait. Have concerns like this already been addressed in more current releases? Do these issues I have seen HP-UX concerns for other operating systems, or should I just continue to modify the standard syslog-ng code as there would be little to no benefit for others? Thanks, /James

2 1

solaris package
by Joop Boonen 28 Jul '06

28 Jul '06

Hello All, I've just build the solaris package via solbuild/rules. When i was doing this i was wondering about a few things. These is a nice init.d file in the solbuild directory. solbuild/syslog-ng.init.d After the build it cant be found in the package at all. When i copy the above mentioned init.d script it uses the config file /opt/syslog-ng/etc/syslog-ng/syslog-ng.conf. Would /etc/syslog-ng/syslog-ng.com a better location? I adapted the script for this now. Why is the package in stalled in /opt/syslog-ng in stead of /usr/local? By the way Sun uses /opt/sfw for gnu software. Why are to init.d packages available in the source, being solbuild/syslog-ng.init.d and contrib/init.d.solaris . The first one is realted to /opt/syslog-ng the second one to /usr/local . Regards, Joop Boonen.

2 3