September 2006 - syslog-ng - lists.balabit.hu

Growing log files (i.e. access.log) and regex.
by Robert.Becker＠motoristsgroup.com 08 Mar '07

08 Mar '07

Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************

6 5

Analysis of logs and alerting
by Muath Al Khalaf 11 Feb '07

11 Feb '07

Hi, I am implementing a syslog-ng centralized correlation server. The server receives the log entries from some servers, apps and devices. Is there any good product (commercial or open-source) with GUI or web interface to analyze these logs and give real time alerts with strong data mining tools? I need a tool which could be used by both help desk non-technical staff and technical engineers. It should give basic alerting on applications' or devices' failures first for help desk, and then engineers could dig deeper into these alerts to understand the problem. Kind regards

4 3

Re: syslog-ng Digest, Vol 15, Issue 12
by dwarnock＠harbrook.net 28 Nov '06

28 Nov '06

I am on holiday from the 17th August returning on Monday the 31st August. If your query is urgent please contact the office on +44 (0)1423 508060 or Neil Harvey on +44 (0)15395 64734 (nharvey(a)harbrook.net) Regards David Warnock

5 9

Syslog-ng load testing
by Vaibhav Goel 22 Nov '06

22 Nov '06

Hi all I am planning to do some capacity and load testing our syslog-ng servers over TCP. Towards that end, I will be writing a script to send packets to a syslog-ng host. Has anyone done anything similar? I was just thinking of writing a simple script that will open multiple sockets to the server and send a message (based upon our current calculated average message size). Any other suggestions are appreciated. Thanks,

4 15

ignore src{}; contents at startup?
by Brian A. Seklecki 25 Oct '06

25 Oct '06

All: On FreeBSD, /dev/klog provides for output from the kernel (normally redirected to /dev/console before syslog starts). At startup, /etc/rc.d/dmesg copies the kernel boot hardware paramters to /var/run/dmesg.boot. If I have my src(); set to: source src { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; ...at starup, syslog-ng drains the contents of /dev/klog. By default it tsends to them to user.notice. I like to log{} these to an SMS pager (so I know if a RAID is failing or other hardware problem). However, I'd like to avoid duplication of the efforts of /etc/rc.d/dmesg on behalf of syslog-ng(8) ...especially since I route user.notice to my pager, and kernel boot dmesg(8) can grow over 50 lines. /dev/klog really isn't a socket. It's a special char device. Is there anyway to tell syslog-ng(8) to ignore it's contents at startup? If it was tail(1), i'd say "tail -0 -f /file". Perhaps clear out the contents of /dev/klog somehow? TIA, ~BAS

2 2

bug report
by Matt Zagrabelny 09 Oct '06

09 Oct '06

i sent out an email a couple of days ago asking about this. i will try once more... does anyone on the list know the preferred way of submitting bug reports to the developers? i have considered using reportbug and letting the debian maintainer facilitate my bug. thanks, -- Matt Zagrabelny - mzagrabe(a)d.umn.edu - (218) 726 8844 University of Minnesota Duluth Information Technology Systems & Services PGP key 1024D/84E22DA2 2005-11-07 Fingerprint: 78F9 18B3 EF58 56F5 FC85 C5CA 53E7 887F 84E2 2DA2 He is not a fool who gives up what he cannot keep to gain what he cannot lose. -Jim Elliot

4 8

program() doesn't restart if it dies
by Hari Sekhon 09 Oct '06

09 Oct '06

I have an email alert script written in python to send me security alerts depending on criteria I have defined. I was working on something else and wanted to clean up so I did a pkill -9 python without remembering that I'm on the syslog server. Of course this killed the alert script, which I noticed didn't restart and syslog-ng kept running without it. So I tested it by creating a condition to generate a log and therefore an alert and I didn't get the email. Not very good. Syslog-ng didn't restart the script defined in program() in syslog-ng.conf, even when it had more input for it, which has gotten me thinking, what happens if the script really dies or is accidentally killed? Syslog-ng must have just dropped the output on the floor or something. The only way I can see to restart it is to restart syslog-ng itself. What is the best way to make this more resilient? I'm currently thinking a cron job to test for the python script and then restart the server if it is not found. Anybody got anything better? -h -- Hari Sekhon

3 4

RE: [syslog-ng] Re: question regarding program name logs
by Justin Randall 07 Oct '06

07 Oct '06

Sorry I pasted the wrong "log" statement, here it is: log { source(src_udp); source(src_tcp); filter(fltr_PROGRAM); destination(dst_ PROGRAM); flags(final, flow-control); }; -----Original Message----- From: Justin Randall Sent: Wednesday, September 27, 2006 10:07 AM To: Syslog-ng users' and developers' mailing list Subject: RE: [syslog-ng] Re: question regarding program name logs Hi and thanks for the response. This is actually relating to UDP logging. I have devices sending Syslog messages via UDP to a central Syslog-NG server. Usually the devices log messages to Syslog-NG via UDP in bursts (about 20-25 message within a couple seconds). The source, filter, destination, log statements for the handling are as follows: source src_udp { udp( ip(IPADDRESS) port(PORT) ); }; filter fltr_PROGRAM { program("PROGRAM") or match("PROGRAM"); }; destination dst_PROGRAM { file( "/var/log/PROGRAM/messages" owner(root) group(tomcat) perm(0640) ); }; log { source(src_udp); source(src_tcp); destination(dst_remote_system); flags(final, flow-control); }; What ends up happening when a host falls into this flow, the first log message is missed. The format of the bursts of messages are exactly the same. This did not happen in 2.0rc2, I have downgraded and verified this. Let me know if there's any other info I can provide that would help out. Regards, Justin. -----Original Message----- From: G.W. Haywood [mailto:ged@jubileegroup.co.uk] Sent: Wednesday, September 27, 2006 7:42 AM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] Re: question regarding program name logs Hi there, On Wed, 27 Sep 2006 Justin Randall wrote: > Can anyone confirm if the patch that was to fix this issue was included > in 2.0rc3? Yes, it was. Note that it only affects logging via UDP. -- 73, Ged. _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 2

Re: question regarding program name logs
by G.W. Haywood 07 Oct '06

07 Oct '06

Hi there, On Wed, 27 Sep 2006 Justin Randall wrote: > Can anyone confirm if the patch that was to fix this issue was included > in 2.0rc3? Yes, it was. Note that it only affects logging via UDP. -- 73, Ged.

2 1

RE: [syslog-ng] Re: question regarding program name logs
by Justin Randall 07 Oct '06

07 Oct '06

After upgrading and monitoring, the logs which would previously get logged incorrectly due to the angle bracket prefixing are now not being logged at all. These logs don't come in with any peculiarity about them. The only thing if interest to note is that it is always the first log message in a group of log messages (i.e. host sends 5 Syslog messages, the first one is missed). What kind of logs, information would be helpful to diagnose? I'm currently running on 2.0rc3. Regards, Justin. -----Original Message----- From: G.W. Haywood [mailto:ged@jubileegroup.co.uk] Sent: Tuesday, September 12, 2006 6:27 AM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] Re: question regarding program name logs Hi there, On Tue, 12 Sep 2006 Justin Randall wrote: > I am currently using Syslog-NG version 2.0rc1. > > I've noticed that sometimes when Syslog-NG is logging messages, it > prefixes something onto the program name which disrupts filters > while matching against a program name. For example, in some logs > for a program named "foo" I'll see logs in the form of "<15>foo" > which fail to be matched against a filter of 'program("foo") '. > This seems to only be noticed on logs which originate from remote > hosts. > > Could anyone provide an explanation for what is going on with this? I reported this problem a couple of weeks ago, and Balazs kindly provided a patch which fixed it. I imagine the patch is in .rc2, try upgrading. -- 73, Ged. _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1