November 2005 - syslog-ng - lists.balabit.hu

Growing log files (i.e. access.log) and regex.
by Robert.Becker＠motoristsgroup.com 08 Mar '07

08 Mar '07

Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************

6 5

ignore src{}; contents at startup?
by Brian A. Seklecki 25 Oct '06

25 Oct '06

All: On FreeBSD, /dev/klog provides for output from the kernel (normally redirected to /dev/console before syslog starts). At startup, /etc/rc.d/dmesg copies the kernel boot hardware paramters to /var/run/dmesg.boot. If I have my src(); set to: source src { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; ...at starup, syslog-ng drains the contents of /dev/klog. By default it tsends to them to user.notice. I like to log{} these to an SMS pager (so I know if a RAID is failing or other hardware problem). However, I'd like to avoid duplication of the efforts of /etc/rc.d/dmesg on behalf of syslog-ng(8) ...especially since I route user.notice to my pager, and kernel boot dmesg(8) can grow over 50 lines. /dev/klog really isn't a socket. It's a special char device. Is there anyway to tell syslog-ng(8) to ignore it's contents at startup? If it was tail(1), i'd say "tail -0 -f /file". Perhaps clear out the contents of /dev/klog somehow? TIA, ~BAS

2 2

Pix problem
by Damien Michau 28 Dec '05

28 Dec '05

Hi All ! I have some probleme ta log my pix's log into my syslog-ng server . i have mounted a Syslog-ng server to store my pix's log . But there is nothing in my pix.log i'have put this lines in my syslog-ng.conf source pix { udp(ip(10.60.10.111) port());}; destination pix { file("/var/log/pix.log"); }; log { source(pix);destination(pix); }; Have you one idea ? Thx Damien Michau Paris

4 5

(2nd attempt) syslog-ng and FreeBSD jails
by markzero 30 Nov '05

30 Nov '05

Hi, Still having problems getting logs out of my FreeBSD jails. No errors are recorded, no clue as to why there are no logs. My config looks like this: --- # $Id: syslog-ng.conf,v 1.3 2005/11/25 11:24:02 mzero Exp $ # CUSTOM # shell config options { owner("syslogng"); group("syslogng"); dir_owner("syslogng"); dir_group("syslogng"); dir_perm(0750); perm(0640); sync(0); stats(1200); create_dirs(yes); keep_hostname(yes); chain_hostnames(no); use_time_recvd(yes); time_reopen(3); use_dns(no); }; #---------------------------------------------------------------------- source log_main { unix-dgram("/var/run/log" owner("syslogng") group("syslogng") perm(0600)); unix-dgram("/var/spool/postfix/dev/log" owner("syslogng") group("syslogng") perm(0600)); unix-dgram("/usr/jail/hosts/ssh/root/var/run/log" owner("syslogng") group("syslogng") perm(0600)); unix-dgram("/usr/jail/hosts/comms/root/var/run/log" owner("syslogng") group("syslogng") perm(0600)); }; source log_internal { internal(); }; #---------------------------------------------------------------------- # chrooted, relative paths destination local_main { file("/local/$YEAR/$MONTH/$DAY-main.log"); }; destination local_internal { file("/local/$YEAR/$MONTH/$DAY-syslog.log"); }; destination tcp_stunnel { tcp("127.0.0.1" localip(127.0.0.1) port(10514) tcp-keep-alive(yes)); }; #---------------------------------------------------------------------- log { source(log_main); destination(local_main); destination(tcp_stunnel); }; log { source(log_internal); destination(local_internal); destination(tcp_stunnel); }; --- The process runs chrooted to /var/log and gives no errors. Logs appear correctly from the host machine, but not from any of the jails. Anybody know what's wrong, is this a bug? M -- pgp: http://www.darklogik.org/pub/pgp/pgp.txt 0160 A46A 9A48 D3B0 C92F B690 17FB 4B72 0207 ED43

2 2

How to uninstall libol and syslog-ng
by Krishnakumar Menon 30 Nov '05

30 Nov '05

Hi, I have installed libol and syslog-ng. Now I would like to know the roll back procedure ie unistalling it. Thanks Krishnakumar __________________________________ Yahoo! Mail - PC Magazine Editors' Choice 2005 http://mail.yahoo.com

2 1

Re: syslog-ng in a freeBSD jail
by markzero 30 Nov '05

30 Nov '05

> >I am running FreeBSD 6.0-RELEASE. I have syslog-ng running on the > >host system and have specified in the config that it should create > >a log socket within the jail (an actual FreeBSD jail(), not a plain > >chroot): > > > > unix-dgram("/usr/jail/hosts/ssh/root/var/run/log" > > owner("syslogng") > > group("syslogng") > > perm(0600)); > > > >The socket is created successfully, I recieve no errors but I also > >recieve no logs from processes running in the jail. Do I also need > >a syslog-ng process running inside the jail? > > No, but you do need to configure your syslog-ng process to read from > "/usr/jail/hosts/ssh/root/var/run/log". Sorry, should have given a bit more detail there really. My config looks like this: --- # $Id: syslog-ng.conf,v 1.3 2005/11/25 11:24:02 mzero Exp $ # CUSTOM # shell config options { owner("syslogng"); group("syslogng"); dir_owner("syslogng"); dir_group("syslogng"); dir_perm(0750); perm(0640); sync(0); stats(1200); create_dirs(yes); keep_hostname(yes); chain_hostnames(no); use_time_recvd(yes); time_reopen(3); use_dns(no); }; #---------------------------------------------------------------------- source log_main { unix-dgram("/var/run/log" owner("syslogng") group("syslogng") perm(0600)); unix-dgram("/var/spool/postfix/dev/log" owner("syslogng") group("syslogng") perm(0600)); unix-dgram("/usr/jail/hosts/ssh/root/var/run/log" owner("syslogng") group("syslogng") perm(0600)); unix-dgram("/usr/jail/hosts/comms/root/var/run/log" owner("syslogng") group("syslogng") perm(0600)); }; source log_internal { internal(); }; #---------------------------------------------------------------------- # chrooted, relative paths destination local_main { file("/local/$YEAR/$MONTH/$DAY-main.log"); }; destination local_internal { file("/local/$YEAR/$MONTH/$DAY-syslog.log"); }; destination tcp_stunnel { tcp("127.0.0.1" localip(127.0.0.1) port(10514) tcp-keep-alive(yes)); }; #---------------------------------------------------------------------- log { source(log_main); destination(local_main); destination(tcp_stunnel); }; log { source(log_internal); destination(local_internal); destination(tcp_stunnel); }; --- Is there some glaring omission here that I'm not aware of? The above looks to me like it should work properly. thanks, mark -- pgp: http://www.darklogik.org/pub/pgp/pgp.txt 0160 A46A 9A48 D3B0 C92F B690 17FB 4B72 0207 ED43

1 0

syslog-ng in a freeBSD jail
by markzero 29 Nov '05

29 Nov '05

Hello. I am running FreeBSD 6.0-RELEASE. I have syslog-ng running on the host system and have specified in the config that it should create a log socket within the jail (an actual FreeBSD jail(), not a plain chroot): unix-dgram("/usr/jail/hosts/ssh/root/var/run/log" owner("syslogng") group("syslogng") perm(0600)); The socket is created successfully, I recieve no errors but I also recieve no logs from processes running in the jail. Do I also need a syslog-ng process running inside the jail? Is there a better solution to this problem? thanks, mark -- pgp: http://www.darklogik.org/pub/pgp/pgp.txt 0160 A46A 9A48 D3B0 C92F B690 17FB 4B72 0207 ED43

2 1

hostname disappear (L4 switch log)
by 김유균 29 Nov '05

29 Nov '05

Hi~ when I use syslog for collecting logs (from network device Route, L2 switch, L4 switch, ) there are no prolbems. => raw log(which is shown in network device) Nov 22 18:46:48 [11.114.116.15.1.2] NOTICE WebOS : real server 211.11.11.111 operational =>log that syslog get Nov 22 18:46:48 [11.114.116.15.1.2] NOTICE WebOS : real server 211.11.11.111 operational nochange!!!!!!!!!!!!!!!!!!!!!! --------------------------- however when I use syslog-ng for collecting logs, the raw log(above) has changed like this~ Nov 22 18:46:48 NOTICE WebOS : real server 211.11.11.111 operational [11.114.116.15.1.2] gone!!!!!!!!!! I think syslog-ng ommit "[11.114.116.15.1.2]" area. Between network device and syslog-ng server , there are no syslog, syslog-ng servers. Do you have same problem? plz help me i use Linux EL4, syslog-ng 1.6.9 <http://www.korea.com> <http://event.korea.com/event/2005/10/fallbooks/>

1 0

syslog-ng, hostname disappear (L4 switch log)
by 김유균 29 Nov '05

29 Nov '05

Hi~ when I use syslog for collecting logs (from network device Route, L2 switch<Cisco>, L4 switch<Alteon>, ) there are no prolbems. => raw log(which is shown in network device) Nov 22 18:46:48 [11.114.116.15.1.2] NOTICE WebOS <slb>: real server 211.11.11.111 operational =>log that syslog get Nov 22 18:46:48 [11.114.116.15.1.2] NOTICE WebOS <slb>: real server 211.11.11.111 operational nochange!!!!!!!!!!!!!!!!!!!!!! --------------------------- however when I use syslog-ng for collecting logs, the raw log(above) has changed like this~ Nov 22 18:46:48 NOTICE WebOS <slb>: real server 211.11.11.111 operational [11.114.116.15.1.2] gone!!!!!!!!!! I think syslog-ng ommit "[11.114.116.15.1.2]" area. Between network device and syslog-ng server , there are no syslog, syslog-ng servers. Do you have same problem? plz help me i use Linux EL4, syslog-ng 1.6.9 <http://www.korea.com> <http://event.korea.com/event/2005/11/dite/>

1 0

Max TCP Connections
by Philip Webster 27 Nov '05

27 Nov '05

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 G'day, I'm currently running syslog-ng 1.6.5 on Red Hat 2.1AS (syslog-ng will be upgraded to 1.6.9 shortly) as a central log server, and I'm having a problem with a growing number of TCP connections. Most hosts log to the central log server via UDP, but there are 7 remote syslog-ng servers using TCP. I currently allow up to 250 connections: source src_tcp { tcp(ip(x.x.x.x) port(514) max-connections(250)); }; but this is reached within about a fortnight. I restart the daemon each week to ensure that it doesn't max out, but figure that there must be a better way. I haven't seen any specific fixes in the last few releases, so is there a particular option I need? Our network is reasonably reliable so I don't think that this is the problem, but there's always a chance ... Any thoughts? Phil -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.7 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDhoegT2yvGjk/8+MRAoeGAJ42YH4BtcjnIxoggrynxesEeHpp4gCeJT19 gOe3dRE69P3IlPI7BdV39m8= =Pbix -----END PGP SIGNATURE-----

2 2