May 2006 - syslog-ng - lists.balabit.hu

Hi everybody i'm new to the list and i have a nice gift, a little script to translate configuration :o)
by Harry Rüter 13 May '06

13 May '06

Hi, i'm simply happy about syslog-ng :o) I coded a shellscript to convert configuration from old syslog to syslog-ng. Works well for me on Suse LINUX 9.1 Have a look at it and enjoy. Would be nice if it would be included in the distribution, but it's tested with 1.6.11 only, so it necessary to test it on different OS and distributions isn't it ? Greets Harry #!/bin/sh ############################################################################### # # Translator from syslog.conf to syslog-ng.conf # by Harry Rueter <h.rueter(a)gmx.net> # # Writes translated configuration to stdout # # New file has to be checked, no warranty at all !!!!!! # Templates are my prefered format. # Some options and sources are commented out # for manual configuration. # Trailing "-" in destination file will not be translated # # History: 12.05.2006: initial coding # ############################################################################### INFILE="$1" if [ "a$INFILE" = "a" ] then echo "error, usage: `basename $0` syslog_configuration_file [> syslog-ng__configuration_file]" exit 1 fi trenner() { echo "########################################################" } trennerecho() { trenner echo "" } trennercomment() { COMMENT="$1" trenner echo "# $COMMENT" trenner } header() { trenner echo "# Generated by `basename $0`(coded by Harry Rueter 5/2006" trenner echo echo "#######################" echo "# Global options #" echo "#######################" echo "options" echo "{" echo " sync(0);" echo " time_reopen(10);" echo " log_fifo_size(100);" echo " #mark(0);" echo " #log_fifo_size();" echo " #chain_hostnames(no);" echo " #create_dirs (no);" echo " #dir_perm(0755);" echo " #dns_cache(yes);" echo " #keep_hostname(yes);" echo " #log_msg_size(8192);" echo " #long_hostnames(on);" echo " #perm(0644);" echo " #stats(3600);" echo " #use_dns(yes);" echo " #use_fqdn(yes);" echo "};" trennerecho } downstr() { echo "$1"|sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' } mk_filter_rules() { INPUT="$1" FAC=`echo "$INPUT"|cut -d"." -f1|sed 's/=//g'` LEVEL=`echo "$INPUT"|cut -d"." -f2|sed 's/=//g'` TMPFAC="" TMPLEVEL="" if [ "$FAC" != "*" ] then TMPFAC="facility($FAC)" fi if [ "$LEVEL" = "*" ] then TMPLEVEL="level(debug..warn)" elif [ "$LEVEL" != "none" ] then TMPLEVEL="level($LEVEL)" fi if [ -n "$TMPLEVEL" -a -n "$TMPFAC" ] then echo "($TMPFAC and $TMPLEVEL)" else echo "($TMPFAC$TMPLEVEL)" fi } mk_dev_log_src() { trennercomment "src=/dev/log" echo "source dev_log_src" echo "{" echo " unix-stream(\"/dev/log\");" echo "};" trennerecho } mk_unix_dgram_log_src() { trennercomment "src=" echo "#source unix_dgram_log_src" echo "#{" echo "# unix-dgram(\"\");" echo "#}:" trennerecho }; mk_udp_log_src() { trennercomment "src=udp" echo "#source udp_log_src" echo "#{" echo "# udp(" echo "# ip(0.0.0.0)" echo "# port(5000)" echo "# max-connections(300)" echo "# );" echo "#};" trennerecho }; mk_tcp_log_src() { trennercomment "src=tcp" echo "#source tcp_log_src" echo "#{" echo "# tcp(" echo "# ip(0.0.0.0)" echo "# port(5000)" echo "# max-connections(300)" echo "# );" echo "#};" trennerecho }; mk_pipe_log_src() { trennercomment "src=pipe" echo "#source pipe_log_src" echo "#{" echo "# pipe();" echo "#};" trennerecho }; mk_internal_log_src() { trennercomment "src=internal()" echo "source internal_src" echo "{" echo " internal();" echo "};" trennerecho } mk_log_src() { mk_dev_log_src mk_unix_dgram_log_src mk_udp_log_src mk_tcp_log_src mk_pipe_log_src mk_internal_log_src echo } mk_dest_file() { FILE="$1" NAME=`basename $FILE|sed 's/\./_/g'` trennercomment "destination file=$FILE" echo "destination d_$NAME" echo "{" echo " file(\"$FILE\"" echo " owner(\"root\")" echo " group(\"root\")" echo " perm(0600)" echo " template( \"[\$YEAR/\$MONTH/\$DAY \$HOUR:\$MIN:\$SEC] \$PROGRAM - \$PRIORITY : \$MSGONLY\n\")" echo " #template_escape(no)" echo " #sync_freq(1000)" echo " #create_dirs(yes/no)" echo " #dir_perm(755)" echo " #dir_owner(root)" echo " #dir_group(root)" echo " );" echo "};" } mk_dest_pipe() { FILE="$1" NAME=`basename $FILE|sed 's/\./_/g'` trennercomment "destination pipe=$FILE" echo "destination d_$NAME" echo "{" echo " pipe(\"$FILE\"" echo " owner(\"root\")" echo " group(\"root\")" echo " perm(0600)" echo " template( \"[\$YEAR/\$MONTH/\$DAY \$HOUR:\$MIN:\$SEC] \$PROGRAM - \$PRIORITY : \$MSGONLY\n\")" echo " #template_escape(no)" echo " );" echo "};" } mk_dest_filter() { NAME="$1" FILTER="$2" echo "filter f_$NAME" echo "{" echo " $FILTER;" echo "};" trenner } mk_log_entry() { FILTER="$1" DEST="$2" LINE="$3" echo "log" echo "{" echo " source(dev_log_src);" echo " filter($FILTER);" echo " destination($DEST);" echo "};" trennerecho } mk_internal_entry() { DEST="/var/log/syslog-ng-internal.log" trennercomment "internal syslog-ng logging" mk_dest_file $DEST echo "log" echo "{" echo " source(internal_src);" echo " destination(d_syslog-ng-internal_log};" echo "};" trennerecho } mk_dest() { DEST="$1" echo $DEST|grep "^|" >/dev/null if [ $? -eq 0 ] then DEST=`echo "$DEST"|sed 's/^|//g'` mk_dest_pipe $DEST else mk_dest_file $DEST fi } proceedline() { LINE="$1" DEST="$2" CNT=1 THEFILTER="" trenner echo "# syslogd: $LINE" echo "# log entry for $DEST" trennerecho mk_dest "$DEST" trenner echo "$LINE"|grep ";" >/dev/null 2>&1 if [ $? -eq 0 ] then while [ 0 -eq 0 ] do PART=`echo "$LINE"|cut -d";" -f$CNT` CNT=`expr $CNT + 1` if [ "a$PART" = "a" ] then NAME="`basename $DEST|sed 's/\./_/g'`" mk_dest_filter "$NAME" "$THEFILTER" break else TMPFILTER=`mk_filter_rules "$PART"` if [ -n "$THEFILTER" ] then THEFILTER="$THEFILTER or $TMPFILTER" else THEFILTER="$TMPFILTER" fi fi done else NAME=`basename $DEST|sed 's/\./_/g'` TMPFILTER=`mk_filter_rules "$LINE"` mk_dest_filter "$NAME" "$TMPFILTER" fi } proceed() { LINE="$1" RULES=`echo "$LINE"|cut -d" " -f1` DEST=`echo "$LINE"|cut -d" " -f2-10|sed 's/^-//g'` if [ "$DEST" = "*" ] then DEST="/var/log/everything.log" fi proceedline "$RULES" "$DEST" WHAT=`basename $DEST|sed 's/\./_/g'` mk_log_entry "f_$WHAT" "d_$WHAT" $LINE } ############################################# # main program ############################################# # statical entrys header echo mk_log_src echo mk_internal_entry echo # dynamical entries cat "$INFILE" |grep -v "^#"|grep -v "^$" |expand|sed 's/ */ /'| \ while read LINE do proceed "$LINE" done

1 0

RE: [syslog-ng] syslog-ng GUI
by Dukes Clayton 11 May '06

11 May '06

Uh, ok... So can someone dumb those pdf's down so that someone with only a 150iq can understand them? Geeze man. I'm working on a development version of php-syslog-ng that does (I think) what sisyphus is referring to. Essentially, group multiple, repeated tokens into the DB thereby decreasing large amounts of data. When a token gets repeated, simply update a count field using "ON DUPLICATE KEY UPDATE count = count + 1" (available in MySQl v4.1+) I just need the time to re-write the tables. I've already done it with the hosts table: I created a new table called hosts and just update the count field, then reference that host using a foreign key in the logs table. -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Jon Stearley Sent: Thursday, May 11, 2006 10:50 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng GUI On May 11, 2006, at 6:51 AM, Arya, Manish Kumar wrote: > Hi Guys, > > I am storing logs on a central server having 3T SAN, using follwing > template > > destination indexlog { > > file("/logs/log01/indexlog/$YEAR/$MONTH/$DAY/$HOST" > > template("$HOUR:$MIN:$SEC,$PROGRAM,$FACILITY,$PRIORITY,$MSGONLY\n") > template-escape(yes) > owner(root) group(root) perm(0644) > dir_perm(0755) create_dirs(yes)); > }; > > my logging is done perfectly :) > > like /logs/log01/indexlog/2006/05/11/hostnames > > I want to have a GUI to view logs with following facilities > > -search logs on basis on date/time, text patterns in > messages,hostnames. http://www.cs.sandia.gov/sisyphus/ mines patterns, but does not have a production GUI (yet). It is more of a research tool at this point, but I would be happy to help you give it a try. Recent emphasis has been on the functionality described in .../detection.pdf. Please let me know if interested, like I said I'd be happy to help, and am in fact looking for additional datasets to analyze; I find my approach to be effective for supercomputer logs, but have not yet explored its effectiveness for other log sets (eg enterprise). I've been waiting to implement a production GUI until I am confident that the underlying functionality is general and excellent. My current leaning is towards adding sisyphus functionality to splunk's interface (and have contacted splunk about this). G'day! -- +--------------------------------------------------------------+ | Jon Stearley (505) 845-7571 (FAX 844-9297) | | Sandia National Laboratories Scalable Systems Integration | +--------------------------------------------------------------+ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

RE: [syslog-ng] syslog-ng GUI
by Dukes Clayton 11 May '06

11 May '06

Try Php-syslog-ng -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Arya, Manish Kumar Sent: Thursday, May 11, 2006 7:51 AM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] syslog-ng GUI Hi Guys, I am storing logs on a central server having 3T SAN, using follwing template destination indexlog { file("/logs/log01/indexlog/$YEAR/$MONTH/$DAY/$HOST" template("$HOUR:$MIN:$SEC,$PROGRAM,$FACILITY,$PRIORITY,$MSGONLY\n") template-escape(yes) owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; my logging is done perfectly :) like /logs/log01/indexlog/2006/05/11/hostnames I want to have a GUI to view logs with following facilities -search logs on basis on date/time, text patterns in messages,hostnames. -should provide facility to have filters associated with user authentication/authorization. -should be able to to parallel search to improve search response time. Regards, -Manish __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1

Multi-threading Syslog-ng
by jarrod.schous＠sasktel.sk.ca 10 May '06

10 May '06

Is syslog-ng capable of multi-threading?. I.E. Can it distribute its processing requirements accross multiple CPU's. I have seen a couple side notes that say it isn't but I was wondering if there is a definitive answer. Thanks Jarrod. NOTICE: This confidential e-mail message is only for the intended recipient(s). If you are not the intended recipient, be advised that disclosing, copying, distributing, or any other use of this message, is strictly prohibited. In such case, please destroy this message and notify the sender.

2 1

cnnection 0 desn't have any source
by wayne yu 10 May '06

10 May '06

Hi everyone: I got waning message when I starting my syslog : Warning: connection 0 doesn't have any sources and I don't how to fixing it. can someone look at my /etc/syslog-ng.conf and tell me how to fixing it. # Options options { use_fqdn(yes); sync(0); keep_hostname(yes); chain_hostnames(no); create_dirs(yes); }; # Sources of syslog messages (both local and remote messages on the server) source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; source s_stunnel { tcp(ip("127.0.0.1") port(514) max-connections(1)); }; source s_udp { udp(); }; # Level Filters filter f_emerg { level (emerg); }; filter f_alert { level (alert .. emerg); }; filter f_crit { level (crit .. emerg); }; filter f_err { level (err .. emerg); }; filter f_warning { level (warning .. emerg); }; filter f_notice { level (notice .. emerg); }; filter f_info { level (info .. emerg); }; filter f_debug { level (debug .. emerg); }; # Facility Filters filter f_kern { facility (kern); }; filter f_user { facility (user); }; filter f_mail { facility (mail); }; filter f_daemon { facility (daemon); }; filter f_auth { facility (auth); }; filter f_syslog { facility (syslog); }; filter f_lpr { facility (lpr); }; filter f_news { facility (news); }; filter f_uucp { facility (uucp); }; filter f_cron { facility (cron); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; filter f_local2 { facility (local2); }; filter f_local3 { facility (local3); }; filter f_local4 { facility (local4); }; filter f_local5 { facility (local5); }; filter f_local6 { facility (local6); }; filter f_local7 { facility (local7); }; # Custom Filters filter f_user_none { not facility (user); }; filter f_kern_debug { filter (f_kern) and filter (f_debug); }; filter f_daemon_notice { filter (f_daemon) and filter (f_notice); }; filter f_mail_crit { filter (f_mail) and filter (f_crit); }; filter f_mesg { filter (f_kern_debug) or filter (f_daemon_notice) or filter (f_mail_crit); }; filter f_authinfo { filter (f_auth) or program (sudo); }; destination hosts { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY$YEAR$MONTH$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); }; #Destinations: local files, the console, and the client files destination l_authlog { file ("/var/log/authlog"); }; destination l_messages { file ("/var/log/messages"); }; destination l_maillog { file ("/var/log/maillog"); }; destination l_ipflog { file ("/var/log/ipflog"); }; destination l_imaplog { file ("/var/log/imaplog"); }; destination l_syslog { file ("/var/log/syslog"); }; destination l_console { file ("/dev/console"); }; destination r_authlog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/authlog"); }; destination r_messages { file ("/var/log/clients/$YEAR/$MONTH/$HOST/messages"); }; destination r_maillog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/maillog"); }; destination r_ipflog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/ipflog"); }; destination r_imaplog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/imaplog"); }; destination r_console { file ("/var/log/clients/$YEAR/$MONTH/$HOST/consolelog"); }; destination r_syslog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/syslog"); }; destination r_fallback { file ("/var/log/clients/$YEAR/$MONTH/$HOST/$FACILITY-$LEVEL"); }; destination mail-alert-perl { program("/usr/local/bin/syslog-mail-perl"); }; filter f_attack_alert { match("attackalert"); }; # find messages with "attackalert" in them, and send to the mail-alert script log { #source(src); filter(f_attack_alert); destination(mail-alert-perl); }; # Log statements # Local sources log { source (s_local); filter (f_authinfo); destination (l_authlog); }; log { source (s_local); filter (f_mail); destination (l_maillog); }; log { source (s_local); filter (f_local0); destination (l_ipflog); }; log { source (s_local); filter (f_local1); destination (l_imaplog); }; log { source (s_local); filter (f_syslog); destination (l_syslog); }; log { source (s_local); filter (f_emerg); filter (f_user_none); destination (l_console); }; log { source (s_local); filter (f_mesg); filter (f_user_none); destination (l_messages); }; # All sources, since we want to archive local and remote logs log { source (s_local); source (s_stunnel); filter (f_authinfo); destination (r_authlog); }; log { source (s_local); source (s_stunnel); filter (f_mail); destination (r_maillog); }; log { source (s_local); source (s_stunnel); filter (f_local0); destination (r_ipflog); }; log { source (s_local); source (s_stunnel); filter (f_local1); destination (r_imaplog); }; log { source (s_local); source (s_stunnel); filter (f_syslog); destination (r_syslog); }; log { source (s_local); source (s_stunnel); filter (f_emerg); filter (f_user_none); destination (l_console); }; log { source (s_local); source (s_stunnel); filter (f_mesg); filter (f_user_none); destination (l_messages); }; Thanks in advance. Wayne Yu __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1

syslog-ng.conf
by wayne yu 08 May '06

08 May '06

Hi : I've been spending 3 weeks on configuring syslog-ng.conf, can some show me a syslog-ng.conf for reference? Thanks Wayne Yu __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

Defunct processes when doing log rotation
by Dev Man 08 May '06

08 May '06

Hi, I'm using syslog-ng 1.9.9 In syslog-ng.conf, I have specified a program to process the syslog-ng output. destination df_mail { file("/var/log/mail.log"); program("/home/readlog/readlog.php"); }; When cron performs it's logrotation daily, it runs a reload of the syslog-ng system. The readlog.php script ends up being <defunct>. For example: root@thehost:/etc/logrotate.d# ps -ef | grep read root 9 1 0 Mar27 ? 00:00:00 [kthread] root 1652 537 0 Apr20 ? 00:00:00 [readlog.php] <defunct> root 4000 537 0 Apr21 ? 00:00:00 [readlog.php] <defunct> root 7189 537 0 Apr22 ? 00:00:00 [readlog.php] <defunct> Thanks for any suggestions. D

3 3

"not netmask(...)" not working
by Brian Candler 08 May '06

08 May '06

Hello, I am using syslog-ng-1.6.11 under RHEL 4, using an RPM built from http://www.balabit.com/downloads/syslog-ng/1.6/src/syslog-ng-1.6.11.tar.gz [*] I tried to use the following rule to match all packets except those from certain host ranges: destination estate { file("/logs/estate.log" ); }; filter f_estate { not netmask( "10.1.0.0/16" ) and not netmask( "172.17.0.0/24" ) and not netmask( "192.168.1.254" ); }; log { source(s_sys); filter(f_estate); destination(estate); }; but it didn't work. tcpdump showed packets from outside those ranges were being received, but syslog-ng did not log them. Now, inspecting the code, firstly it seems I should have written the expressions in dotted netmask form, i.e. "10.1.0.0/255.255.0.0". It would be nice if syslog-ng were to validate this a bit better :-) However when I fixed that nothing changed. Looking in the code, I think that a negation operation is missing for the netmask() function, perhaps something like this: --- src/filters.c.orig Fri May 5 13:19:18 2006 +++ src/filters.c Fri May 5 13:24:44 2006 @@ -272,10 +272,10 @@ netw = self->network.s_addr; mask = self->netmask.s_addr; - return ((host & mask) == (netw & mask)); + return ((host & mask) == (netw & mask)) ^ c->comp; } else { - return 0; + return c->comp; } } However, I don't understand why some of the functions use c->comp, and others use self->super.comp, so I'm a bit hesitant to modify in this way. I was able to workaround by rewriting the expression thus: destination estate { file("/logs/estate.log" ); }; filter f_not_estate { netmask( "10.1.0.0/255.255.0.0" ) or netmask( "172.17.0.0/255.255.255.0" ) or netmask( "192.168.1.254" ); }; filter f_estate { not filter("f_not_estate"); }; log { source(s_sys); filter(f_estate); destination(estate); }; but perhaps someone who understands the expression parser could look at fixing this properly? Thanks, Brian. [*] I had to use syslog-ng.spec.bb, because syslog-ng.spec references a number of non-existent files, such as Source3: mysql-syslog.pipe Source4: mysql-syslog.buffer) With a small change to syslog-ng.bb, changing an underscore to dash, it was happy.

2 1

syslog-ng blocking on read() from /proc/kmsg
by John Morrissey 07 May '06

07 May '06

I'm running the Debian 1.6.9-1 package with the 'read from each file descriptor n times or until EAGAIN' patch backported. There seems to be a problem with this behavior when it's applied to /proc/kmsg. Here's the source we use: source s_kmsg { file("/proc/kmsg" log_prefix("kernel: ")); }; What happens is: 1. syslog-ng calls poll(2) and waits. 2. Data appears on /proc/kmsg and causes poll() to return. 3. syslog-ng read(2)s from /proc/kmsg, grabbing some data. 4. Since it hasn't received EAGAIN or read n times from this file descriptor, syslog-ng happily read()s again. Since it's exhausted the available data, the read() blocks. Forever. 5. Meanwhile, other syslog traffic to /dev/log piles up. Eventually, whatever buffer there is on Unix domain sockets fills up and causes other applications on the machine to block, waiting to syslog. 6. The box eventually runs out of file descriptors since other processes (cron is a big culprit because it runs jobs so often) are piling up, holding onto file descriptors and blocking on writes to /dev/log. The blocking behavior on /proc/kmsg is interesting, since if I'm reading the source correctly, file sources are set O_NONBLOCK: affile.c: 152 static int do_init_affile_source(struct log_handler *c, struct syslog_con 153 { 154 CAST(affile_source, self, c); 155 int fd, flags; 156 157 if (self->flags & AFFILE_PIPE) 158 flags = O_RDWR | O_NOCTTY | O_NONBLOCK | O_LARGEFILE; 159 else 160 flags = O_RDONLY | O_NOCTTY | O_NONBLOCK | O_LARGEFILE; Any thoughts? thanks, john -- John Morrissey _o /\ ---- __o jwm(a)horde.net _-< \_ / \ ---- < \, www.horde.net/ __(_)/_(_)________/ \_______(_) /_(_)__

2 2

syslog-ng 1.6.11 released
by Balazs Scheidler 06 May '06

06 May '06

Hi, I have released syslog-ng version 1.6.11, available at http://www.balabit.com/products/syslog-ng/ The user visible changes this time: News for the 1.6.11 release Wed, 03 May 2006 13:08:38 +0200 * Using time_sleep() caused a parser error as a rule was missed from the configuration parser. * Scalability improvements applied in 1.6.10 broke reading /proc/kmsg on Linux. -- Bazsi

1 0