January 2006 - syslog-ng - lists.balabit.hu

Growing log files (i.e. access.log) and regex.
by Robert.Becker＠motoristsgroup.com 08 Mar '07

08 Mar '07

Here's the scenario. We have a JBoss server that outputs to *.log files. After doing all the research I could, I found out the only way to get these logs sent to syslog-ng was through a tail & logger command. The problem with that is logger appends the date and some other information to that logged message. When attempting to put that information in a mysql database I'm getting errors. So, what I would like to do is use some form of regex to alter the statement so it can be inserted into a database. There are basically two things that need to be changed. First, logger appends the date/time it receives the message and it also says it is coming from logger. I would like to remove that date and switch logger to jboss or something similar. Here is an example of what I am doing. This is a line from the jboss log file: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Then I send the log through logger to syslog-ng: #!/bin/sh tail -f /opt/jboss/server/default/log/boot.log | logger -p local7.info The log gets turned into this: Oct 11 13:14:15 src@linuxp17 logger: 2005-10-11 13:14:15,848 DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 What I need is: Oct 11 13:14:15 src@linuxp17 jboss: DEBUG [org.jboss.resource.connectionmanager.IdleRemover] run: IdleRemover notifying pools, interval: 450000 Now, I've thought of using awk in the tail/logger script to remove the date, but that does not take care of the logger/jboss substitution. Anyone have any ideas? Can syslog-ng do this with the match() function? -Rob Becker Systems Engineer Motorists Insurance ********************************************************************** The information contained in this message is confidential and is intended for the addressee(s) only. If you have received this message in error or there are any problems please notify the originator immediately. The unauthorized use, disclosure, copying or alteration of this message is strictly forbidden. Motorists Insurance Group will not be liable for direct, special, indirect or consequential damages arising from the alteration of the contents of this message by a third party or as a result of any virus being passed on. **********************************************************************

6 5

ignore src{}; contents at startup?
by Brian A. Seklecki 25 Oct '06

25 Oct '06

All: On FreeBSD, /dev/klog provides for output from the kernel (normally redirected to /dev/console before syslog starts). At startup, /etc/rc.d/dmesg copies the kernel boot hardware paramters to /var/run/dmesg.boot. If I have my src(); set to: source src { unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); internal(); file("/dev/klog"); }; ...at starup, syslog-ng drains the contents of /dev/klog. By default it tsends to them to user.notice. I like to log{} these to an SMS pager (so I know if a RAID is failing or other hardware problem). However, I'd like to avoid duplication of the efforts of /etc/rc.d/dmesg on behalf of syslog-ng(8) ...especially since I route user.notice to my pager, and kernel boot dmesg(8) can grow over 50 lines. /dev/klog really isn't a socket. It's a special char device. Is there anyway to tell syslog-ng(8) to ignore it's contents at startup? If it was tail(1), i'd say "tail -0 -f /file". Perhaps clear out the contents of /dev/klog somehow? TIA, ~BAS

2 2

"MSG_DONTWAIT undeclared"-error during compiling libol-0.3.17 on HP-UX 11.00/11.11
by Tob_Sch＠gmx.de 22 Mar '06

22 Mar '06

Hi, compiling libol-0.3.16 (and syslog-ng-1.6.8) on HP-UX 11.00/11.11 worked without any problems. Today, I tried to compile libol-0.3.17 (and syslog-ng-1.6.9). I got following errors during 'make': Making all in src make all-am source='io.c' object='io.lo' libtool=yes \ depfile='.deps/io.Plo' tmpdepfile='.deps/io.TPlo' \ depmode=hp /bin/sh /tmp/libol-0.3.17/depcomp \ /bin/sh ../libtool --mode=compile cc -DHAVE_CONFIG_H -I. -I/tmp/libol-0.3.17/src -I. -D_GNU_SOURCE -D_XOPEN_SOURCE_EXTENDED -g -Wall -O2 -c -o io.lo `test -f 'io.c' || echo '/tmp/libol-0.3.17/src/'`io.c cc -DHAVE_CONFIG_H -I. -I/tmp/libol-0.3.17/src -I. -D_GNU_SOURCE -D_XOPEN_SOURCE_EXTENDED -g -Wall -O2 -c io.c -Wp,-M.deps/io.TPlo +Z -DPIC -o .libs/io.o cc: "io.c", line 493: error 1588: "MSG_DONTWAIT" undefined. cc: "io.c", line 493: warning 563: Argument #4 is not the correct type. *** Error exit code 1 Stop. *** Error exit code 1 Stop. *** Error exit code 1 Stop. It seems, that there is no MSG_DONTWAIT variable defined in the socket.h file on HP-UX like for example on SUSE 10.0. It seems, that the reference to MSG_DONTWAIT came new with libol-0.3.17. -- Lust, ein paar Euro nebenbei zu verdienen? Ohne Kosten, ohne Risiko! Satte Provisionen f�r GMX Partner: http://www.gmx.net/de/go/partner

4 8

io.c: do_read: read( ) failed (errno 1), Operation not permitted
by Rick.Daoust＠borealc.on.ca 01 Feb '06

01 Feb '06

I installed syslog-ng version 1.6.9 on Fedora Core 4 in a chroot jail and when starting it, I receive the following message "io.c: do_read: read( ) failed (errno 1), Operation not permitted". I believe this message is the result of syslog-ng not being able to read from /proc/kmsg. However, I'm a newbie to Linux and Sylog-ng! Any suggestions would be greatly appreciated!

4 4

Inserting text in syslog messages
by Gustavo Mendes de Carvalho 31 Jan '06

31 Jan '06

Dear All, Does anybody knows how to insert some string in a message that will be apended in log files ? I mean, imagine that syslog-ng receives this message to insert in /var/log/auth.log. Jan 31 13:18:01 host sshd[6278]: Accepted keyboard-interactive/pam for user from 10.10.0.29 port 49105 ssh2 And I would like to insert some text together, resulting in some like this Jan 31 13:18:01 host sshd[6278]: Accepted keyboard-interactive/pam for root from 10.10.0.29 port 49105 ssh2 - service xyz for user root How can I include this string " - service xyz for user root" ? I know that it's possible, but I didn't find how to. Thanks in advance Gustavo

2 1

SuSE 10.0 - Changing permissions on special file /dev/xconsole
by George 30 Jan '06

30 Jan '06

Hello, I am getting the following in my /var/log/warn file at each bootup. What is it about and how do I fix it? Jan 26 06:26:52 dns1 syslog-ng[3430]: Changing permissions on special file /dev/xconsole Jan 26 06:26:52 dns1 syslog-ng[3430]: Changing permissions on special file /dev/tty10 I have SuSE 10.0 installed on a IBM X330 (8654-51Y) eserver xseries with a dual Pentium III (x84) and a S3 Savage 4 video card. Thanks, George -- George genz1968(a)fastmail.fm -- http://www.fastmail.fm - One of many happy users: http://www.fastmail.fm/docs/quotes.html

2 3

Newbie
by ajc80＠o2.co.uk 29 Jan '06

29 Jan '06

Hello, I'm a newbie to syslog-ng, I have followed the instructions and used and altered the necessary examples to suit my directory structures, so my syslog messages are dumped in to mysql database. I'm using: syslog-ng-1.6.8 libol-0.3.16 mysql 3.23.58 When starting syslog-ng I get the following messages: # /etc/init.d/syslog-ng start Starting MySQL pipe... Starting syslog-ng... Warning: No source refers to internal messages, they'll go to /dev/null unresolved reference: inputs Error initializing configuration, exiting. The "No source refers to internal messages...." message I understand. The "unresolved reference: inputs" I don't and can't seem to find anything in the archives. I've included the syslog-ng.conf file and the mysql.pipe file below. Any help would be appreciated. Thanks, Adrian. ************ /etc/init.d/mysql.pipe *********** #!/bin/sh # # Script to create a named pipe so syslog-ng messages # can be sent to a MySQL database # if [ -p /var/run/mysql.pipe ]; then mysql -u xxxxxxx --password=xxxxxxx logs < /var/run/mysql.pipe & else mkfifo /var/run/mysql.pipe mysql -u xxxxxx --password=xxxxxx logs < /var/run/mysql.pipe & fi ************ /etc/syslog-ng/syslog-ng.conf *********** ##################################################################### # syslog-ng central log host configuration file # # installed: 29/01/2006 13:18 # ##################################################################### # General options options { check_hostname(yes); keep_hostname(yes); chain_hostnames(no); }; # Other important general options # use_fqdn(yes); # sync(0); # keep_hostname(yes); # chain_hostnames(no); # create_dirs(yes); ##################################################################### # Define syslog input stream # This is specific to Solaris 9 # source inputs { internal(); # sun-streams("/dev/log" door("/var/run/syslog_door")); # tcp(max_connections(1000)); }; # For newer versions of Linux (Debian GNU/Linux woody, # and RedHat Linux post 7.0) # source stdlog { unix-dgram("/dev/log"); }; ##################################################################### # Define filters # Level Filters filter f_emerg { level (emerg); }; filter f_alert { level (alert .. emerg); }; filter f_crit { level (crit .. emerg); }; filter f_err { level (err .. emerg); }; filter f_warning { level (warning .. emerg); }; filter f_notice { level (notice .. emerg); }; # Facility Filters filter f_kern { facility (kern); }; filter f_user { facility (user); }; filter f_mail { facility (mail); }; filter f_daemon { facility (daemon); }; filter f_auth { facility (auth); }; filter f_syslog { facility (syslog); }; filter f_lpr { facility (lpr); }; filter f_news { facility (news); }; filter f_uucp { facility (uucp); }; filter f_cron { facility (cron); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; filter f_local2 { facility (local2); }; filter f_local3 { facility (local3); }; filter f_local4 { facility (local4); }; filter f_local5 { facility (local5); }; filter f_local6 { facility (local6); }; filter f_local7 { facility (local7); }; # syslog-ng stats filter filter f_n_stats { not match("STATS: dropped 0"); }; # syslog-ng "connect from filter filter f_n_connected { not match("AF_INET client connected from"); }; ##################################################################### # Define where the logs can be stored # Log log files destination logpile { file("/var/log/syslog-ng/$HOST/$YEAR/$MONTH/$FACILITY.$YEAR$MONTH$DAY" owner(root) group(root) perm(0600) create_dirs(yes) dir_perm(0700)); }; # Console log destination console { file("/dev/sysmsg"); }; # Linux Console # destination console { file("/dev/console"); }; # MySQL Database destination d_mysql { pipe("/var/run/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL','$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG' );\n") template-escape(yes)); }; ##################################################################### # Define logging # Local log log { source(inputs); filter(f_n_stats); filter(f_notice); filter(f_n_connected); destination(logpile); }; # Console log { source(inputs); filter(f_n_stats); filter(f_err); filter(f_n_connected); destination(console); }; # MySQL Database log { source(inputs); filter(f_n_stats); filter(f_notice); filter(f_n_connected); destination(d_mysql); }; ________________________________________ For a wide range of mobile phones,free upgrades,accessories and tariffs plus special deals when you buy online,visit www.o2.co.uk/shop <http://www.o2.co.uk/shop>

2 1

Crosscompile Syslog-ng
by Kenneth Kassing 27 Jan '06

27 Jan '06

I am crosscompileing syslog-ng to to a powerpc, and had to make a few changes to make it compile properly. If anyone is interested I have included these two patch files here. First I updated configure.in to allow you to specify a path to libfl.a (it should also work with libl.a, but I have not tested this). The configure tests already included does not find the lex libraries for me (I do not have them installed. Instead I am just building flex as compile time dependency.) Second, the configure script was explicitly running configure and make from the script. I did not like configure calling make, and it was not using any of my configure arguements for libol. As a result libol was built for my build machine instead of the target device. The syslog-ng configure script will now now call configure for libol (with all the supplied options), and will not run make. When make is run from the syslog-ng directory it will also run make for libol. -Ken --- Makefile.am.org 2006-01-24 17:18:51.000000000 -0500 +++ Makefile.am 2006-01-24 22:26:21.000000000 -0500 @@ -1,4 +1,4 @@ -SUBDIRS = src doc contrib +SUBDIRS = $(LIBOL_DIR) src doc contrib EXTRA_DIST = PORTS syslog-ng.spec syslog-ng.spec.in syslog-ng.spec.bb syslog-ng.spec.bb.in README.spoof dist.conf dist.conf.in --- configure.in.org 2006-01-24 11:56:15.000000000 -0500 +++ configure.in 2006-01-25 11:19:34.000000000 -0500 @@ -56,6 +56,11 @@ AC_ARG_WITH(libnet, [ --with-libnet=path use path to libnet-config script], , [with_libnet=""]) + +AC_ARG_WITH(lex, + AC_HELP_STRING([--with-lexlib=path],[use an uninstalled copy of lex]), + [with_lex=`cd $with_lex; pwd`], + [with_lex=""]) AC_ARG_ENABLE(sun-streams, [ --enable-sun-streams enable Sun STREAMS support even if not detected]) @@ -85,8 +90,8 @@ sysconfdir="${sysconfdir}/syslog-ng" AM_SANITY_CHECK dnl Checks for programs. -AC_PROG_CC -AM_PROG_CC_STDC +#AC_PROG_CC +#AM_PROG_CC_STDC AC_PROG_YACC AM_PROG_LEX AC_PROG_MAKE_SET @@ -355,15 +360,33 @@ else AC_MSG_RESULT([no clues, linking everything dynamically, please send appropriate ld arguments to syslog-ng(a)lists.balabit.hu] fi -AC_MSG_CHECKING(libol version >= $LIBOL_REQ_MAJOR.$LIBOL_REQ_MINOR.$LIBOL_REQ_PATCH) +if test "$with_lex" != ""; then + AC_MSG_CHECKING(for uninstalled version of lex in $with_lex) + if test -f "$with_lex/libfl.a"; then + LEXLIB="$with_lex/libfl.a" + AC_MSG_RESULT($LEXLIB) + elif test -f "$with_lex/libl.a"; then + LEXLIB="$with_lex/libl.a" + AC_MSG_RESULT($LEXLIB) + else + AC_ERROR(no) + fi +fi + +AC_MSG_CHECKING(for libol) if test "$with_libol" = ""; then if test ! -x "`which libol-config`" ; then bundled_libol_dir="`find $srcdir/libol-* -maxdepth 0 -type d 2>/dev/null`" if test -n "$bundled_libol_dir"; then - test -d $bundled_libol_dir || mkdir $bundled_libol_dir - (cd $bundled_libol_dir; ./configure && make) + test -d $bundled_libol_dir || mkdir $bundled_libol_dir with_libol=$bundled_libol_dir + AC_MSG_RESULT(Configuration required) + + # Remove the single quotes from the configure args, and pass them to + # configure for libol + ac_configure_str=`echo $ac_configure_args | sed -e "s/'//g"` + (cd $bundled_libol_dir; ./configure $ac_configure_str) else AC_ERROR(Bundled libol not found) fi @@ -372,9 +395,12 @@ if test "$with_libol" = ""; then LIBOL_CFLAGS=`libol-config --cflags` LIBOL_LIBS=`libol-config --static-libs` MAKE_CLASS="make_class" + AC_MSG_RESULT(ok) fi fi +AC_MSG_CHECKING(libol version >= $LIBOL_REQ_MAJOR.$LIBOL_REQ_MINOR.$LIBOL_REQ_PATCH) + if test "$with_libol" != ""; then if test ! -x $with_libol/libol-config; then AC_ERROR(libol-config not found in the specified location) @@ -386,7 +412,6 @@ if test "$with_libol" != ""; then MAKE_CLASS="$with_libol/utils/make_class" fi - LIBOL_VERSION=[`echo $LIBOL_VERSION | cut -d '+' -f 1`] LIBOL_MAJOR=[`echo $LIBOL_VERSION | cut -d '.' -f 1`] LIBOL_MINOR=[`echo $LIBOL_VERSION | cut -d '.' -f 2`]

1 0

Syslog-ng with Ipsec or with Stunnel?
by Mounib Khanafer 27 Jan '06

27 Jan '06

Hi all, I hope if someone can help me with this. I'm researching syslog-ng and colllecting as much info as posiible bout it. I want to know which does perform better when working with syslog-ng: IPsec or Stunnel? Please note that I use Solaris 8 and 9 worksations. Thanks a lot. Best Regards, Mounib -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Heigl Florian - Munich-MR - external Sent: Friday, January 27, 2006 11:43 AM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] chroot or configuration issue when directories getcreated as 'root'? Hi list, I'm stumbling over what is hopefully just a configuration issue... syslog-ng 'seems' to drop privileges and run as user syslogng, but creates new directories as root and will fail to write new logfiles inside them. I'm afraid this is related to still having port 514 open... Funny thing: The files in those directories will use the specified permissions :) I was starting syslog-ng with the following parameters: /opt/syslog-ng/sbin/syslog-ng --pidfile=/var/run/syslog-ng.pid --chroot=/var/opt/syslog-ng --user=syslogng --group=syslogng # Log file options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (on); use_dns (yes); use_fqdn (yes); create_dirs (yes); # leave this to 'no', see http://www.campin.net/syslog-ng/faq.html#bad_filenames keep_hostname (no); }; # Source - only listen on UDP, don't read /dev/log source s_udp { udp(); }; # [ I cut filters section out ] # testing external input now destination logip { file("/logs/remote/$HOST_FROM/$YEAR$MONTH/$FACILITY$YEAR$MONTH$DAY" owner(syslogng) group(syslogng) perm(0600) dir_perm(0700) create_dirs(yes) template("$DATE $FULLHOST $PROGRAM $TAG [$FACILITY.$LEVEL] $MESSAGE\n") ); }; log { source(s_udp); destination(logip); }; # EOF Currently I'm running it as root and am thinking about using a portforward 514->10514 so I don't require any root privileges. Alternatively I could try out Balabit's restrict, which seems to be better at dropping root privileges, but unfortunately I have no clue how it works, and am not sure if it will work on HP-UX :) Can You try to make me understand where the problem is hiding? Configuration? Port Number? Bug? User? Thanks a lot of Your time! Florian -- I'll save You the signature :) _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

chroot or configuration issue when directories get created as 'root'?
by Heigl Florian - Munich-MR - external 27 Jan '06

27 Jan '06

Hi list, I'm stumbling over what is hopefully just a configuration issue... syslog-ng 'seems' to drop privileges and run as user syslogng, but creates new directories as root and will fail to write new logfiles inside them. I'm afraid this is related to still having port 514 open... Funny thing: The files in those directories will use the specified permissions :) I was starting syslog-ng with the following parameters: /opt/syslog-ng/sbin/syslog-ng --pidfile=/var/run/syslog-ng.pid --chroot=/var/opt/syslog-ng --user=syslogng --group=syslogng # Log file options { sync (0); time_reopen (10); log_fifo_size (1000); long_hostnames (on); use_dns (yes); use_fqdn (yes); create_dirs (yes); # leave this to 'no', see http://www.campin.net/syslog-ng/faq.html#bad_filenames keep_hostname (no); }; # Source - only listen on UDP, don't read /dev/log source s_udp { udp(); }; # [ I cut filters section out ] # testing external input now destination logip { file("/logs/remote/$HOST_FROM/$YEAR$MONTH/$FACILITY$YEAR$MONTH$DAY" owner(syslogng) group(syslogng) perm(0600) dir_perm(0700) create_dirs(yes) template("$DATE $FULLHOST $PROGRAM $TAG [$FACILITY.$LEVEL] $MESSAGE\n") ); }; log { source(s_udp); destination(logip); }; # EOF Currently I'm running it as root and am thinking about using a portforward 514->10514 so I don't require any root privileges. Alternatively I could try out Balabit's restrict, which seems to be better at dropping root privileges, but unfortunately I have no clue how it works, and am not sure if it will work on HP-UX :) Can You try to make me understand where the problem is hiding? Configuration? Port Number? Bug? User? Thanks a lot of Your time! Florian -- I'll save You the signature :)

1 0