May 2006 - syslog-ng - lists.balabit.hu

Growing log files (i.e. access.log) and regex.
by Robert.Becker＠motoristsgroup.com 08 Mar '07

08 Mar '07

Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************

6 5

Analysis of logs and alerting
by Muath Al Khalaf 11 Feb '07

11 Feb '07

Hi, I am implementing a syslog-ng centralized correlation server. The server receives the log entries from some servers, apps and devices. Is there any good product (commercial or open-source) with GUI or web interface to analyze these logs and give real time alerts with strong data mining tools? I need a tool which could be used by both help desk non-technical staff and technical engineers. It should give basic alerting on applications' or devices' failures first for help desk, and then engineers could dig deeper into these alerts to understand the problem. Kind regards

4 3

ignore src{}; contents at startup?
by Brian A. Seklecki 25 Oct '06

25 Oct '06

All: On FreeBSD, /dev/klog provides for output from the kernel (normally redirected to /dev/console before syslog starts). At startup, /etc/rc.d/dmesg copies the kernel boot hardware paramters to /var/run/dmesg.boot. If I have my src(); set to: source src { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; ...at starup, syslog-ng drains the contents of /dev/klog. By default it tsends to them to user.notice. I like to log{} these to an SMS pager (so I know if a RAID is failing or other hardware problem). However, I'd like to avoid duplication of the efforts of /etc/rc.d/dmesg on behalf of syslog-ng(8) ...especially since I route user.notice to my pager, and kernel boot dmesg(8) can grow over 50 lines. /dev/klog really isn't a socket. It's a special char device. Is there anyway to tell syslog-ng(8) to ignore it's contents at startup? If it was tail(1), i'd say "tail -0 -f /file". Perhaps clear out the contents of /dev/klog somehow? TIA, ~BAS

2 2

filter on level don't work with kernel source
by Olleg Samoylov 14 Jul '06

14 Jul '06

$ syslog-ng --version syslog-ng 1.9.9 subj from syslog-ng.conf: source s_all { # message generated by Syslog-NG internal(); # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" log_prefix("kernel: ")); }; filter f_at_least_crit { level(crit..emerg); }; log { source(s_all); filter(f_at_least_crit); destination(d_email2root); }; And I get emails like: Date: 2006-04-05T14:59:23+04:00 Facility: kern Level: debug Program: kernel ISO 9660 Extensions: Microsoft Joliet Level 3 Date: 2006-04-05T14:59:23+04:00 Facility: kern Level: debug Program: kernel ISO 9660 Extensions: Microsoft Joliet Level 3 -- Olleg Samoylov

2 2

RE: [syslog-ng] Starting syslog-ng as root
by Muath Al Khalaf 02 Jun '06

02 Jun '06

Thank you very much. I have disabled SELinux and every thing goes fine. Do you think it is better to run syslog-ng with or without SELinux especially that I may use MySQL and PHP-syslog-ng? Kind regards -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Jose Pedro Oliveira Sent: Saturday, May 27, 2006 6:59 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Starting syslog-ng as root -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Muath Al Khalaf wrote: > Hi, > I am using Redhat Enterprise 4. I am using the official rpm image built > by Balabit for RHE 4 with their startup script. The executable did not > return anything (at least in front of me inside console). For strace I > do not know how to use it. You may be having problems with SELinux. In RHEL4, CentOS, and Fedora Core 3 you need to enable the use_syslogng SELinux boolean before starting the syslog-ng daemon [1]. To check the use_syslogng boolean status getsebool -a | grep syslogng To enable it (and saving its value) setsebool -P use_syslogng 1 jpo [1] - you need to have the a recent selinux-policy-targeted - -- José Pedro Oliveira * mailto: jpo(a)di.uminho.pt * http://gsd.di.uminho.pt/jpo * * gpg fingerprint = F9B6 8D87 859D 1C94 48F0 84C0 9749 9EB5 91BD 851B * -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iD8DBQFEeHc6l0metZG9hRsRAvqLAJ41iuwCrKNhFhyG2lWyv6Q2eScdzACgmoMD QzNXnXkc4tuHT0bbK5Fl5iw= =s/7p -----END PGP SIGNATURE-----

4 7

1.9.11 compile failure on IRIX 6.5, Tru64 UNIX 5.1
by Albert Chin 01 Jun '06

01 Jun '06

Seems like IRIX 6.5 and Tru64 UNIX 5.1 don't have IPV6_ADD_MEMBERSHIP. They do have IP_ADD_MEMBERSHIP though. Should we just: #ifndef IPV6_ADD_MEMBERSHIP #define IPV6_ADD_MEMBERSHIP IP_ADD_MEMBERSHIP #endif cc -DHAVE_CONFIG_H -I. -I. -I.. -DPATH_SYSCONFDIR=\"/etc/opt/TWWfsw/syslogng1911\" -DPATH_LOCALSTATEDIR=\"/var/opt/TWWfsw/syslogng1911\" -I/opt/TWWfsw/libglib26/include -I/opt/TWWfsw/libglib26/lib/include -I/opt/TWWfsw/gettext014/include -I/opt/TWWfsw/syslogng198/include/eventlog -D_GNU_SOURCE -Wl,-woff,84 -Wl,-woff,85 -woff 1009,1429 -use_readonly_const -G0 -rdata_shared -DEBUG:optimize_space=ON -O2 -OPT:Olimit=0 -c afinet.c cc-1020 cc: ERROR File = afinet.c, Line = 190 The identifier "IPV6_ADD_MEMBERSHIP" is undefined. setsockopt(fd, IPPROTO_IPV6, IPV6_ADD_MEMBERSHIP, &mreq6, sizeof(mreq6)); ^ 1 error detected in the compilation of "afinet.c". -- albert chin (china(a)thewrittenword.com)

2 2

1.9.11 compile failure on HP-UX 11.23 and AIX 5.x
by Albert Chin 01 Jun '06

01 Jun '06

I'm getting the same compile failure on HP-UX and AIX using the vendor compiler: xlc -DHAVE_CONFIG_H -I. -I. -I.. -DPATH_SYSCONFDIR=\"/etc/opt/TWWfsw/syslogng1911\" -DPATH_LOCALSTATEDIR=\"/var/opt/TWWfsw/syslogng1911\" -I/opt/TWWfsw/libglib26/include -I/opt/TWWfsw/libglib26/lib/include -I/opt/TWWfsw/gettext014/include -I/opt/TWWfsw/syslogng1911/include/eventlog -D_GNU_SOURCE -O2 -ma -qro -qroconst -qmaxmem=-1 -qarch=com -c afinet.c "/usr/include/alloca.h", line 34.1: 1506-224 (I) Incorrect pragma ignored. "afinet.c", line 183.13: 1506-021 (S) Expecting struct or union. gmake[2]: *** [afinet.o] Error 1 gmake[2]: Leaving directory `/opt/build/syslog-ng-1.9.11/src' The problem is the following line: if (IN6_IS_ADDR_MULTICAST(g_sockaddr_inet6_get_address(addr)->s6_addr)) On AIX, from <netinet/in.h>: #define IN6_IS_ADDR_MULTICAST(p) IS_MULTIADDR6(*p) #define IS_MULTIADDR6(a) ((a).s6_addr8[0] == 0xff) #define s6_addr8 u6_addr.u6_addr8 #define s6_addr u6_addr.u6_addr8 So, IN6_IS_ADDR_MULTICAST(addr) is converted to IS_MULTIADDR6(*addr) and then: ((*addr).u6_addr.u6_addr8[0] == 0xff So, because addr == g_sockaddr_inet6_get_address(addr)->s6_addr, we get the following expansion: if (((*g_sockaddr_inet6_get_address(addr)->u6_addr.u6_addr8).u6_addr.u6_addr8[0] == 0xff)) The Redhat Linux definition is: #define IN6_IS_ADDR_MULTICAST(a) (((__const uint8_t *) (a))[0] == 0xff) The HP-UX 11.23 definition is: #define IN6_IS_ADDR_MULTICAST(i) ((((i)->s6_addr[0] ^ 0xff) \ | ((i)->s6_addr[1] & 0xe0)) == 0) ntp-4.2.0a also uses IN6_IS_ADDR_MULTICAST but does the following: if (IN6_IS_ADDR_MULTICAST(&((struct sockaddr_in6*)srcadr)->sin6_addr)) -- albert chin (china(a)thewrittenword.com)

2 2

RE: [syslog-ng] Compiling 1.9.11 on Solaris 9
by Heigl Florian - Munich-MR - external 29 May '06

29 May '06

> Not to mention that IPPROTO_IPV6 as a name suggests the IPv6 is a > subprotocol of IP itself, which is IPv4. This is certainly not true. You're just calling the 'Internet Protocol Version 4' 'IP'. This certainly can be confusing. SCNR, florian

1 0

RE: [syslog-ng] Starting syslog-ng as root
by Muath Al Khalaf 27 May '06

27 May '06

Hi, I am using Redhat Enterprise 4. I am using the official rpm image built by Balabit for RHE 4 with their startup script. The executable did not return anything (at least in front of me inside console). For strace I do not know how to use it. -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Balazs Scheidler Sent: Friday, May 26, 2006 10:35 PM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] Starting syslog-ng as root On Fri, 2006-05-26 at 16:58 +0300, Muath Al Khalaf wrote: > Hi, > > > > I am trying to start syslog-ng as root but failed. The startup script > is using daemon. I have tried also by issuing /sbin/syslog-ng but got > nothing. I know it is better to run it with user privileges but this > is to be able to log kernel messages file("proc/kmsg"); hmm.. I don't see why it failed, but I would certainly need some more information to be able to help you. The return code of the executable, an strace dump or something. -- Bazsi _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1

Compiling 1.9.11 on Solaris 9
by Albert Chin 26 May '06

26 May '06

I tried building syslog-ng 1.9.11 on Solaris 9 with the Sun C compiler. It failed with the following: cc -DHAVE_CONFIG_H -I. -I. -I.. -DPATH_SYSCONFDIR=\"/etc/opt/TWWfsw/syslogng1911\" -DPATH_LOCALSTATEDIR=\"/var/opt/TWWfsw/syslogng1911\" -I/opt/TWWfsw/libglib26/include -I/opt/TWWfsw/libglib26/lib/include -I/opt/TWWfsw/gettext014/include -I/opt/TWWfsw/syslogng198/include/eventlog -D_GNU_SOURCE -mr -Qn -xstrconst -xO2 -xtarget=ultra2 -xarch=v8plusa -c afinet.c "afinet.c", line 81: warning: implicit function declaration: memset "afinet.c", line 156: undefined symbol: SOL_IP "afinet.c", line 162: undefined symbol: SOL_IP "afinet.c", line 169: undefined symbol: SOL_IP "afinet.c", line 172: undefined symbol: SOL_IP "afinet.c", line 181: warning: left operand of "->" must be pointer to struct/union "afinet.c", line 188: undefined symbol: SOL_IPV6 "afinet.c", line 194: undefined symbol: SOL_IPV6 "afinet.c", line 200: undefined symbol: SOL_IPV6 cc: acomp failed for afinet.c gmake[2]: *** [afinet.o] Error 2 gmake[2]: Leaving directory `/opt/build/syslog-ng-1.9.11/src' The prototype for memset() is in <string.h>. Solaris 9 doesn't have SOL_IP anywhere in /usr/include. From setsockopt(3SOCKET) on Solaris 9: When manipulating socket options, the level at which the option resides and the name of the option must be specified. To manipulate options at the "socket" level, level is speci- fied as SOL_SOCKET. To manipulate options at any other level, level is the protocol number of the protocol that controls the option. For example, to indicate that an option is to be interpreted by the TCP protocol, level is set to the TCP protocol number . See getprotobyname(3SOCKET). AIX 5.3 doesn't build for the same reason: "afinet.c", line 157.32: 1506-045 (S) Undeclared identifier SOL_IP. "afinet.c", line 182.13: 1506-021 (S) Expecting struct or union. "afinet.c", line 189.32: 1506-045 (S) Undeclared identifier SOL_IPV6. gmake[2]: *** [afinet.o] Error 1 gmake[2]: Leaving directory `/opt/build/syslog-ng-1.9.11/src' Ditto for HP-UX 11.23/PA and 11.23/IA. -- albert chin (china(a)thewrittenword.com)

2 3