August 2006 - syslog-ng - lists.balabit.hu

Growing log files (i.e. access.log) and regex.
by Robert.Becker＠motoristsgroup.com 08 Mar '07

08 Mar '07

Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************

6 5

Analysis of logs and alerting
by Muath Al Khalaf 11 Feb '07

11 Feb '07

Hi, I am implementing a syslog-ng centralized correlation server. The server receives the log entries from some servers, apps and devices. Is there any good product (commercial or open-source) with GUI or web interface to analyze these logs and give real time alerts with strong data mining tools? I need a tool which could be used by both help desk non-technical staff and technical engineers. It should give basic alerting on applications' or devices' failures first for help desk, and then engineers could dig deeper into these alerts to understand the problem. Kind regards

4 3

Re: syslog-ng Digest, Vol 15, Issue 12
by dwarnock＠harbrook.net 28 Nov '06

28 Nov '06

I am on holiday from the 17th August returning on Monday the 31st August. If your query is urgent please contact the office on +44 (0)1423 508060 or Neil Harvey on +44 (0)15395 64734 (nharvey(a)harbrook.net) Regards David Warnock

5 9

ignore src{}; contents at startup?
by Brian A. Seklecki 25 Oct '06

25 Oct '06

All: On FreeBSD, /dev/klog provides for output from the kernel (normally redirected to /dev/console before syslog starts). At startup, /etc/rc.d/dmesg copies the kernel boot hardware paramters to /var/run/dmesg.boot. If I have my src(); set to: source src { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; ...at starup, syslog-ng drains the contents of /dev/klog. By default it tsends to them to user.notice. I like to log{} these to an SMS pager (so I know if a RAID is failing or other hardware problem). However, I'd like to avoid duplication of the efforts of /etc/rc.d/dmesg on behalf of syslog-ng(8) ...especially since I route user.notice to my pager, and kernel boot dmesg(8) can grow over 50 lines. /dev/klog really isn't a socket. It's a special char device. Is there anyway to tell syslog-ng(8) to ignore it's contents at startup? If it was tail(1), i'd say "tail -0 -f /file". Perhaps clear out the contents of /dev/klog somehow? TIA, ~BAS

2 2

Re: RE: Re: syslog-ng 2.0rc1 memory usage
by Martin, David M 10 Sep '06

10 Sep '06

here are the definitely lost record results. short duration (start valgrind, send hup, wait, send hup, kill) ==7989== 70 bytes in 1 blocks are definitely lost in loss record 15 of 43 ==7989== at 0x1B904984: malloc (vg_replace_malloc.c:131) ==7989== by 0x12AB72: g_malloc (in /usr/lib/libglib-2.0.so.0.400.7) ==7989== by 0x1394F5: g_strdup (in /usr/lib/libglib-2.0.so.0.400.7) ==7989== by 0x80559B6: log_writer_options_init (logwriter.c:507) ==7989== 1080 bytes in 10 blocks are definitely lost in loss record 35 of 43 ==7989== at 0x1B905301: calloc (vg_replace_malloc.c:176) ==7989== by 0x12ABDF: g_malloc0 (in /usr/lib/libglib-2.0.so.0.400.7) ==7989== by 0x80530FD: log_msg_new (logmsg.c:545) ==7989== by 0x805BE42: log_reader_handle_line (logreader.c:178) ==7989== LEAK SUMMARY: ==7989== definitely lost: 1150 bytes in 11 blocks. ==7989== possibly lost: 0 bytes in 0 blocks. ==7989== still reachable: 113453 bytes in 1157 blocks. ==7989== suppressed: 200 bytes in 1 blocks. ------------------------------ Message: 2 Date: Wed, 09 Aug 2006 10:00:02 +0200 From: Balazs Scheidler <bazsi(a)balabit.hu> Subject: Re: [syslog-ng] RE: Re: syslog-ng 2.0rc1 memory usage (Balazs Scheidler) To: Syslog-ng users' and developers' mailing list <syslog-ng(a)lists.balabit.hu> Message-ID: <1155110402.6312.14.camel(a)bzorp.balabit> Content-Type: text/plain On Tue, 2006-08-08 at 15:56 -0500, Martin, David M wrote: > applied the patch and resident size increased only marginally over a > 10 minute period. much better. > ran through valgrind another 10 minutes, here's the report. Only one definitely lost record was present, which should be fixed by the patch below. Some of the remaining leaks seem to be part of buffered messages not yet sent to destinations, but they were still reachable. Some reload tests would be appreciated (e.g. run syslog-ng and reload it a couple of times with your configuration and load), that might reveal some real leaks. --- orig/src/messages.c +++ mod/src/messages.c @@ -53,10 +53,12 @@ msg_send_internal_message(int prio, cons { LogMessage *m; - g_snprintf(buf, sizeof(buf), "<%d> syslog-ng[%d]: %s\n", prio, getpid(), msg); - m = log_msg_new(buf, strlen(buf), NULL, LP_INTERNAL | LP_LOCAL, NULL); if (G_LIKELY(internal_msg_queue)) - g_queue_push_tail(internal_msg_queue, m); + { + g_snprintf(buf, sizeof(buf), "<%d> syslog-ng[%d]: %s\n", prio, getpid(), msg); + m = log_msg_new(buf, strlen(buf), NULL, LP_INTERNAL | LP_LOCAL, NULL); + g_queue_push_tail(internal_msg_queue, m); + } } } -- Bazsi ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 16, Issue 9 **************************************** The information transmitted (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is intended only for the person(s) or entity/entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient(s) is prohibited. If you received this in error, please contact the sender and delete the material from any computer.

2 1

syslog-ng-2.0rc1 doesn't log hostaddress using ipv6
by Bruce Penrod 10 Sep '06

10 Sep '06

I'm using the unpatched syslog-ng-2.0rc1. When logging messages sent from another machine over udp6, the hostname of the sending machine is not logged and the receiving host's name is logged instead, so that it looks like the message was generated locally. When using udp, it works fine. -- Bruce Penrod

2 1

couple install questions
by Nick Baronian 06 Sep '06

06 Sep '06

I have a RedHat Ent.4 box I want to put syslog-ng on. Should I remove syslogd (sysklogd) first? If so how do I get syslog-ng to handle the cron (vixie) and initscripts? How can I find out what configure options are available to me for syslog-ng? I figured I would just try compiling syslog-ng on the machine and see what happens with syslogd. I built glib-2.12.0 & eventlog-0.2.4 and then exported PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig I did a ./configure --enable-tcp-wrapper. But during make I get this error gcc -g -O2 -Wall -g -o syslog-ng main.o libsyslog-ng.a -lnsl -lrt -Wl,-Bstatic -lfl -L/usr/local/lib -lglib-2.0 -L/usr/local/lib -levtlog -Wl,-Bdynamic libsyslog-ng.a(logmsg.o)(.text+0x6a): In function `log_stamp_format': /root/syslog-ng-2.0rc1/src/logmsg.c:111: undefined reference to `g_assert_warning' libsyslog-ng.a(logmsg.o)(.text+0x2c2): In function `log_msg_ref': /root/syslog-ng-2.0rc1/src/logmsg.c:489: undefined reference to `g_assert_warning' libsyslog-ng.a(logmsg.o)(.text+0x35e): In function `log_msg_unref': /root/syslog-ng-2.0rc1/src/logmsg.c:503: undefined reference to `g_assert_warning' libsyslog-ng.a(logmsg.o)(.text+0xd9a): In function `log_msg_ack_block_end': /root/syslog-ng-2.0rc1/src/logmsg.c:623: undefined reference to `g_return_if_fail_warning' libsyslog-ng.a(logpipe.o)(.text+0x4f): In function `log_pipe_ref': /root/syslog-ng-2.0rc1/src/logpipe.c:38: undefined reference to `g_assert_warning' libsyslog-ng.a(logpipe.o)(.text+0x9b): In function `log_pipe_unref': /root/syslog-ng-2.0rc1/src/logpipe.c:47: undefined reference to `g_assert_warning' libsyslog-ng.a(center.o)(.text+0x7d): In function `log_endpoint_free': /root/syslog-ng-2.0rc1/src/center.c:90: undefined reference to `g_assert_warning' libsyslog-ng.a(center.o)(.text+0x175): In function `log_connection_new': /root/syslog-ng-2.0rc1/src/center.c:132: undefined reference to `g_assert_warning' libsyslog-ng.a(templates.o)(.text+0x2a6): In function `log_template_ref': /root/syslog-ng-2.0rc1/src/templates.c:169: undefined reference to `g_assert_warning' libsyslog-ng.a(templates.o)(.text+0x30e):/root/syslog-ng-2.0rc1/src/templates.c:180: more undefined references to `g_assert_warning' follow collect2: ld returned 1 exit status make[2]: *** [syslog-ng] Error 1 make[2]: Leaving directory `/root/syslog-ng-2.0rc1/src' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/root/syslog-ng-2.0rc1' make: *** [all] Error 2 Does anyone have any ideas on what is causing this? Thanks, Nick Baronian

4 3

Installation options
by Anshul Saxena 01 Sep '06

01 Sep '06

Hi, I want to install syslog-ng on my rhel4.0 box. I was going through the "Hardening Linux" book, Chapter 5 , "Understanding Logging and Log monitoring" . It is written there that in order to install syslog-ng, i have to first install the "libol" package . But I cannot find the package anywhere on this page: http://www.balabit.com/products/syslog_ng/upgrades.bbq which has been mentioned in the book. So I haven't started to install syslog-ng as yet. Please let me know the step by step procedure to install syslog-ng and ny dependencies with it. Regards

3 2

Using Syslog NG 2.0-rc1 instead of 1.6.11 for production
by Jochen Kirn 31 Aug '06

31 Aug '06

Hi, we plan to introduce Syslog NG in our environment. When will be version 2.0 final released ? Do you think version 2.0 rc 1 is already stable enough to start with ? thx Jochen

2 1

Pipe to MYSQL does not work
by Matthew Redel 30 Aug '06

30 Aug '06

Thanks for the response. Tailing the pipe while SYSLOG-NG is running shows nothing being input to the pipe. As I mentioned, however, SYSLOG-NG writes to file just fine. The databse is working, I can manually login to the db and do an INSERT just fine. The script does persist and specifies the correct the db, username and password. However, this script is a moot point if SYSLOG-NG doesn't even send to the pipe. Could my Libol version be an issue, or woul;d SYSLOG-NG not even run if Libol is not correct? Matt

2 1