March 2006 - syslog-ng - lists.balabit.hu

Growing log files (i.e. access.log) and regex.
by Robert.Becker＠motoristsgroup.com 08 Mar '07

08 Mar '07

Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************

6 5

ignore src{}; contents at startup?
by Brian A. Seklecki 25 Oct '06

25 Oct '06

All: On FreeBSD, /dev/klog provides for output from the kernel (normally redirected to /dev/console before syslog starts). At startup, /etc/rc.d/dmesg copies the kernel boot hardware paramters to /var/run/dmesg.boot. If I have my src(); set to: source src { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; ...at starup, syslog-ng drains the contents of /dev/klog. By default it tsends to them to user.notice. I like to log{} these to an SMS pager (so I know if a RAID is failing or other hardware problem). However, I'd like to avoid duplication of the efforts of /etc/rc.d/dmesg on behalf of syslog-ng(8) ...especially since I route user.notice to my pager, and kernel boot dmesg(8) can grow over 50 lines. /dev/klog really isn't a socket. It's a special char device. Is there anyway to tell syslog-ng(8) to ignore it's contents at startup? If it was tail(1), i'd say "tail -0 -f /file". Perhaps clear out the contents of /dev/klog somehow? TIA, ~BAS

2 2

TTL patch, helpful for multicasting logs
by Alexander Clouter 04 May '06

04 May '06

Hi, After my earlier multicast patch I quickly realised that if syslog-ng is actually sending UDP multicast syslog data the TTL on the IP packets is by default 1. This results in the packet being expired at the gateway and never being able to leave the subnet....not too handy :( After clobbering it around for a few days I have rolled out this patch to address the problem which should be applied in addition, and after, the multicast patch I released the other day (also re-included in this email). The patch adds a new parameter to the destination tcp()/udp() drivers called ttl() and it takes a number between 0 and 255. When set to zero, it uses the default kernel parameter[1] but when set to 1->255 the TTL on the packet being sent is set to this. Of course if the parameter is not specified it defaults to zero. This should now allow you to use destination du_moo { udp("239.1.2.3" port(514) ttl(7)); }; in your config files. It works on unicast traffic too, I guess you could use it as a layer of defence to guarentee that your syslog packets never leave your network by setting the TTL to a low value. I always wanted to get around to emailing the samba folk a TTL patch to do this sort of thing, so roaming samba laptop users could get some feeling of security that only the local network can get access to their shares....however this is now getting a bit offtopic :) Hope you all find it useful, now back to my multicasting syslog cluster :) Cheers Alex [1] under linux its one, and probably the same for most OS's

2 5

2GB log file size restrictions 1.9 tree
by William Bell 01 Apr '06

01 Apr '06

Is there a way to configure this restriction, nothin is mentioned about it in the doc's. Up until recently I deployed 1.6 tree across all my devices and i never ran into this problem. I just recently upgraded to the 1.9 tree and now syslog-ng just crashes when a 2gb file is reached. If anyone has ANY ideas there are greatly accepted. William Bell CWIE Security

2 1

syslog-ng 1.6.10 released
by Balazs Scheidler 31 Mar '06

31 Mar '06

Hi, As some of you might have read it on my blog (http://bazsi.blogspot.com) I have released syslog-ng 1.6.10, the next stable version in the 1.6.x series, with the following changes: News for the 1.6.10 release Mon, 27 Mar 2006 16:43:08 +0200 * Updated documentation, documentation is now generated from XML instead of SGML. * Scalability improvements for many parallel syslog connections tunable using the time_sleep() global option. -- Bazsi

1 0

syslog-ng & mysql set up for network devices only
by Patrick Topping 31 Mar '06

31 Mar '06

I have a MySQL question that I hope does not sound to 'newbie-ish'..:-) I have just set up a new server for logging all the network devices. Syslog-ng is working fine and now I am ready to move onto setting up the MySQL side. I was wondering if any had created a MySQL database specifically for network devices and if it was significantly different than what is on the following URL: http://vermeer.org/display_doc.php?doc_id=1 CREATE TABLE logs ( host varchar(32) default NULL, facility varchar(10) default NULL, priority varchar(10) default NULL, level varchar(10) default NULL, tag varchar(10) default NULL, date date default NULL, time time default NULL, program varchar(15) default NULL, msg text, seq int(10) unsigned NOT NULL auto_increment, PRIMARY KEY (seq), KEY host (host), KEY seq (seq), KEY program (program), KEY time (time), KEY date (date), KEY priority (priority), KEY facility (facility) ) TYPE=MyISAM; The question that I have relates to following tables in the DB: priority : where would this come into play for network devices? level: not sure what this is tag: where would this come into play for network devices? program: where would this come into play for network devices? If there is anyone who could help me with this network specific DB I would appreciate it. Thanks in advance. -Patrick

1 0

RE: [syslog-ng] Creating a named pipe (FIFO)
by Timothy A. Holmes 29 Mar '06

29 Mar '06

Good morning Peter: thank you for your reply Does the name of the named pipe need to be the same as the destination line mkfifo splunk; destination splunk { pipe("/var/syslog-ng/syslog_fifo"); }; This is what I currently have in the config file now, and I am still getting an error. Thank you TIM My entire syslog-ng.conf is posted at http://www.pastebin.com/616048 Thank you Timothy A. Holmes IT Manager / Network Admin / Web Master / Computer Teacher Medina Christian Academy A Higher Standard... Jeremiah 33:3 Jeremiah 29:11 Esther 4:14 > -----Original Message----- > From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng- > bounces(a)lists.balabit.hu] On Behalf Of HÖLTZL Péter > Sent: Wednesday, March 22, 2006 9:07 AM > To: Syslog-ng users' and developers' mailing list > Subject: Re: [syslog-ng] Creating a named pipe (FIFO) > > On Wed, 2006-03-22 at 08:50 -0500, Timothy A. Holmes wrote: > > I did some investigation and asked a few questions on IRC, and someone > > told me that I needed to use the mkpipe command, however I have been > > unable to locate (using google) the proper syntax for the mkpipe > > command. > > you can create pipes by 'mknod PIPENAME p' command on any linux box. > 'mkfifo PIPENAME' makes the same pipe as well. > > Péter > > > -- > BalaBit IT Bizt. Kft | Tel: +36 1 371-0540 | GnuPG Fingerprint: > holtzl.peter(a)balabit.hu | Mobil: +36 20 366-9667 | 796B C9D3 E492 B006 > C8B2 > http://www.balabit.hu/ | Fax: +36 1 208-0875 | 4D1F 5320 28E3 9A1B > 3FC6

3 2

blog available
by Balazs Scheidler 28 Mar '06

28 Mar '06

Hi, After giving some thought to the idea I decided to start my blog where I'm planning to publish syslog-ng related things as well. If you are interested you can find it at: http://bazsi.blogspot.com/ blogger.com has RSS feeds, so you can use that as well. -- Bazsi

1 0

RE: [syslog-ng] multicast support
by Heigl Florian - Munich-MR - external 28 Mar '06

28 Mar '06

Hi, > Let me know what you think, I wrote this patch to make > clustering logging > servers dead easy and straight forward....I hope its useful > to someone else > too. This will come in handy some day, and even if I can't use it 'right now', thank You lots for taking the time to implement that. florian

1 0

RE: [syslog-ng] prune identical messages
by Richard Legault 28 Mar '06

28 Mar '06

I would prefer the message suppressed if possible as it gives more information, sorry for not being specific. So is it possible to have the suppressed message. As I have not seen any such option? Richard -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu]On Behalf Of Valdis.Kletnieks(a)vt.edu Sent: March 27, 2006 4:42 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] prune identical messages On Mon, 27 Mar 2006 15:26:41 EST, Richard Legault said: > But the message repeating does not give you any new information so it is a wa ste of diskspace to store it. > Because It is just as helpfull to say > foo1: ssh connection from 129.257.10.4 > foo1: 2,348 duplicate messages suppressed *exactly*. But what you asked for was (your words now): "those occurring between would simply be dropped." Producing a "duplicate messages suppressed" is *not* "simply dropping". Also, notice that if you have an interleaved series of 2 or more message streams that are identical to themselves but not each other, you can get this: foo1: message 1 foo1: message 2 foo1: message 3 foo1: message 2 foo1: message 3 foo1: message 1 foo1: message 3 and so on - this requires keeping a copy of the last N messages rather than just the last single one to do it correctly. Proper choice of N to prevent being DoS'ed by an out-of-memory when flooded by a mass of non-identical messages is left as an exercise for the student....

3 4