- syslog-ng - lists.balabit.hu

error installing
by Raymond Norton 17 May '06

17 May '06

Any idea how to fix this? /home/admin/syslog-ng-1.6.10/src/cfg-lex.c:1126: undefined reference to `yywrap' collect2: ld returned 1 exit status make[3]: *** [syslog-ng] Error 1 make[3]: Leaving directory `/home/admin/syslog-ng-1.6.10/src' make[2]: *** [all-recursive] Error 1 make[2]: Leaving directory `/home/admin/syslog-ng-1.6.10/src' make[1]: *** [all] Error 2 make[1]: Leaving directory `/home/admin/syslog-ng-1.6.10/src' make: *** [all-recursive] Error 1

3 3

Newbie Looking for Help
by Mark R. White 16 May '06

16 May '06

I want to setup a syslog-ng server to do syslogging for my network. So far it's easy, the part I'm having trouble with, and I'm not even sure if it's possible, is I want to log by device to it's own individual syslog file. Hence the reason for using syslog-ng vice sysklogd, I think the syntax would be something as follows, but I could be way off base. device1 /var/log/network/device1 log { source(device1) filter (?????) destination (?????) }; I also don't understand where to define the facilities as I did in standard syslog. Any help would be greatly appreciated. One other quick question, when I do a default install of syslog-ng (on Debian Etch), it automatically logs to the tty. Is this correct for all versions, and whether it is or it's not, how do I stop it? Thanks again, Mark

5 9

Help: Cannot open file for writing (Permission denied)
by Muath Al Khalaf 15 May '06

15 May '06

Hi, I have just installed syslog-ng on Redhat enterprise 4. I am using it as remote logging server for a syslog-ng client. Here is my remote logging configuration: source s_remote { tcp(ip(0.0.0.0) port(514)); udp(ip(0.0.0.0) port(514)); }; destination d_separatedbyhosts { file("/var/log/syslog-ng/$HOST/messages" owner("root") group("root") perm(0640) dir_perm(0750) create_dirs(yes)); }; log { source(s_remote); destination(d_separatedbyhosts); }; I am getting the following error message in /var/log/syslog-ng.log on the same machine using tail "cannot open file /var/log/syslog-ng/host1messages for writing (Permission denied)" I have tried to create the file myself and give it 777 permission along with all the directories /var, /var/log, /var/log/syslog-ng If you notice here, 1- syslog-ng is writing error messages from the same machine to /var/log/syslog-ng which means syslog-ng is working well for local logging on the same machine ( I have stopped syslogd). 2- The error has a file name "host1messages" while in the configuration file it is "/$HOST/messages" 3- I am accessing the syslog-ng server using remote SSH from windows using putty client. When I open the /var/log/syslog-ng.log file using "vi" I got a strange character instead of "/" in the previous error message between "host1" and "messages". While - as you have seen above - using tail I do not get any character neither / nor the strange one in the syslog-ng file. Kind regards

1 0

RE: [syslog-ng] Newbie Looking for Help
by Mark R. White 13 May '06

13 May '06

Thanks for all the help I've received so far. Although, I'm still not able to do what I need to do. Right now, I just want to log one specific device to a specific log file. One email I received said I should use the netmask filter option, although for some reason, my system doesn't like that option. Here is the temp syslog-ng.conf I'm using. I have the full backed up, I just want to get this part working. Syslog-ng starts, but it doesn't actually log any thing. I cut this down so it would be easier to work with, although it's very possible that I cut out a required part of the log file. As of right now, I don't care if it logs any from the local system. I just need to log the pix device listed. Thanks in advance for any help, Mark # # Configuration file for syslog-ng under Debian # # attempts at reproducing default syslog behavior # the standard syslog levels are (in descending order of priority): # emerg alert crit err warning notice info debug # the aliases "error", "panic", and "warn" are deprecated # the "none" priority found in the original syslogd configuration is # only used in internal messages created by syslogd ###### # options options { # disable the chained hostname format in logs # (default is enabled) chain_hostnames(0); # the time to wait before a died connection is re-established # (default is 60) time_reopen(10); # the time to wait before an idle destination file is closed # (default is 60) time_reap(360); # the number of lines buffered before written to file # you might want to increase this if your disk isn't catching with # all the log messages you get or if you want less disk activity # (say on a laptop) # (default is 0) #sync(0); # the number of lines fitting in the output queue log_fifo_size(2048); # enable or disable directory creation for destination files create_dirs(yes); # default owner, group, and permissions for log files # (defaults are 0, 0, 0600) owner(root); group(root); perm(0644); # default owner, group, and permissions for created directories # (defaults are 0, 0, 0700) dir_owner(root); dir_group(root); dir_perm(0744); # enable or disable DNS usage # syslog-ng blocks on DNS queries, so enabling DNS may lead to # a Denial of Service attack # (default is yes) use_dns(no); # maximum length of message in bytes # this is only limited by the program listening on the /dev/log Unix # socket, glibc can handle arbitrary length log messages, but -- for # example -- syslogd accepts only 1024 bytes # (default is 2048) #log_msg_size(2048); }; # sources # all known message sources source s_all { # message generated by Syslog-NG internal(); # standard Linux log source (this is the default place for the syslog() # function to send logs to) unix-stream("/dev/log"); # messages from the kernel file("/proc/kmsg" log_prefix("kernel: ")); # use the above line if you want to receive remote UDP logging messages # (this is equivalent to the "-r" syslogd flag) udp(); }; #destinations for log files destination corppix_d { file ("/var/log/network/corppix/pix.log" owner (root) group (root) perm (0644) dir_perm (0744)); }; #filters filter corppix_f { netmask("10.10.10.2"); facility(local4); }; # logs log { filter(corppix_f); destination(corppix_d); };

2 1

Tokenizing messages
by Dukes Clayton 13 May '06

13 May '06

Hi Folks, I'm trying to figure out if there is a way to further parse incoming messages. Right now I have: destination db_mysql { pipe("/var/log/mysql.pipe" template("INSERT INTO logs (host, facility, priority, level, datetime, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC','$MSG' );\n") template-escape(yes)); }; But I want to take $MSG and parse it into individual tokens. How can I accomplish this? Tia! -Clayton

1 0

Hi everybody i'm new to the list and i have a nice gift, a little script to translate configuration :o)
by Harry Rüter 13 May '06

13 May '06

Hi, i'm simply happy about syslog-ng :o) I coded a shellscript to convert configuration from old syslog to syslog-ng. Works well for me on Suse LINUX 9.1 Have a look at it and enjoy. Would be nice if it would be included in the distribution, but it's tested with 1.6.11 only, so it necessary to test it on different OS and distributions isn't it ? Greets Harry #!/bin/sh ############################################################################### # # Translator from syslog.conf to syslog-ng.conf # by Harry Rueter <h.rueter(a)gmx.net> # # Writes translated configuration to stdout # # New file has to be checked, no warranty at all !!!!!! # Templates are my prefered format. # Some options and sources are commented out # for manual configuration. # Trailing "-" in destination file will not be translated # # History: 12.05.2006: initial coding # ############################################################################### INFILE="$1" if [ "a$INFILE" = "a" ] then echo "error, usage: `basename $0` syslog_configuration_file [> syslog-ng__configuration_file]" exit 1 fi trenner() { echo "########################################################" } trennerecho() { trenner echo "" } trennercomment() { COMMENT="$1" trenner echo "# $COMMENT" trenner } header() { trenner echo "# Generated by `basename $0`(coded by Harry Rueter 5/2006" trenner echo echo "#######################" echo "# Global options #" echo "#######################" echo "options" echo "{" echo " sync(0);" echo " time_reopen(10);" echo " log_fifo_size(100);" echo " #mark(0);" echo " #log_fifo_size();" echo " #chain_hostnames(no);" echo " #create_dirs (no);" echo " #dir_perm(0755);" echo " #dns_cache(yes);" echo " #keep_hostname(yes);" echo " #log_msg_size(8192);" echo " #long_hostnames(on);" echo " #perm(0644);" echo " #stats(3600);" echo " #use_dns(yes);" echo " #use_fqdn(yes);" echo "};" trennerecho } downstr() { echo "$1"|sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/' } mk_filter_rules() { INPUT="$1" FAC=`echo "$INPUT"|cut -d"." -f1|sed 's/=//g'` LEVEL=`echo "$INPUT"|cut -d"." -f2|sed 's/=//g'` TMPFAC="" TMPLEVEL="" if [ "$FAC" != "*" ] then TMPFAC="facility($FAC)" fi if [ "$LEVEL" = "*" ] then TMPLEVEL="level(debug..warn)" elif [ "$LEVEL" != "none" ] then TMPLEVEL="level($LEVEL)" fi if [ -n "$TMPLEVEL" -a -n "$TMPFAC" ] then echo "($TMPFAC and $TMPLEVEL)" else echo "($TMPFAC$TMPLEVEL)" fi } mk_dev_log_src() { trennercomment "src=/dev/log" echo "source dev_log_src" echo "{" echo " unix-stream(\"/dev/log\");" echo "};" trennerecho } mk_unix_dgram_log_src() { trennercomment "src=" echo "#source unix_dgram_log_src" echo "#{" echo "# unix-dgram(\"\");" echo "#}:" trennerecho }; mk_udp_log_src() { trennercomment "src=udp" echo "#source udp_log_src" echo "#{" echo "# udp(" echo "# ip(0.0.0.0)" echo "# port(5000)" echo "# max-connections(300)" echo "# );" echo "#};" trennerecho }; mk_tcp_log_src() { trennercomment "src=tcp" echo "#source tcp_log_src" echo "#{" echo "# tcp(" echo "# ip(0.0.0.0)" echo "# port(5000)" echo "# max-connections(300)" echo "# );" echo "#};" trennerecho }; mk_pipe_log_src() { trennercomment "src=pipe" echo "#source pipe_log_src" echo "#{" echo "# pipe();" echo "#};" trennerecho }; mk_internal_log_src() { trennercomment "src=internal()" echo "source internal_src" echo "{" echo " internal();" echo "};" trennerecho } mk_log_src() { mk_dev_log_src mk_unix_dgram_log_src mk_udp_log_src mk_tcp_log_src mk_pipe_log_src mk_internal_log_src echo } mk_dest_file() { FILE="$1" NAME=`basename $FILE|sed 's/\./_/g'` trennercomment "destination file=$FILE" echo "destination d_$NAME" echo "{" echo " file(\"$FILE\"" echo " owner(\"root\")" echo " group(\"root\")" echo " perm(0600)" echo " template( \"[\$YEAR/\$MONTH/\$DAY \$HOUR:\$MIN:\$SEC] \$PROGRAM - \$PRIORITY : \$MSGONLY\n\")" echo " #template_escape(no)" echo " #sync_freq(1000)" echo " #create_dirs(yes/no)" echo " #dir_perm(755)" echo " #dir_owner(root)" echo " #dir_group(root)" echo " );" echo "};" } mk_dest_pipe() { FILE="$1" NAME=`basename $FILE|sed 's/\./_/g'` trennercomment "destination pipe=$FILE" echo "destination d_$NAME" echo "{" echo " pipe(\"$FILE\"" echo " owner(\"root\")" echo " group(\"root\")" echo " perm(0600)" echo " template( \"[\$YEAR/\$MONTH/\$DAY \$HOUR:\$MIN:\$SEC] \$PROGRAM - \$PRIORITY : \$MSGONLY\n\")" echo " #template_escape(no)" echo " );" echo "};" } mk_dest_filter() { NAME="$1" FILTER="$2" echo "filter f_$NAME" echo "{" echo " $FILTER;" echo "};" trenner } mk_log_entry() { FILTER="$1" DEST="$2" LINE="$3" echo "log" echo "{" echo " source(dev_log_src);" echo " filter($FILTER);" echo " destination($DEST);" echo "};" trennerecho } mk_internal_entry() { DEST="/var/log/syslog-ng-internal.log" trennercomment "internal syslog-ng logging" mk_dest_file $DEST echo "log" echo "{" echo " source(internal_src);" echo " destination(d_syslog-ng-internal_log};" echo "};" trennerecho } mk_dest() { DEST="$1" echo $DEST|grep "^|" >/dev/null if [ $? -eq 0 ] then DEST=`echo "$DEST"|sed 's/^|//g'` mk_dest_pipe $DEST else mk_dest_file $DEST fi } proceedline() { LINE="$1" DEST="$2" CNT=1 THEFILTER="" trenner echo "# syslogd: $LINE" echo "# log entry for $DEST" trennerecho mk_dest "$DEST" trenner echo "$LINE"|grep ";" >/dev/null 2>&1 if [ $? -eq 0 ] then while [ 0 -eq 0 ] do PART=`echo "$LINE"|cut -d";" -f$CNT` CNT=`expr $CNT + 1` if [ "a$PART" = "a" ] then NAME="`basename $DEST|sed 's/\./_/g'`" mk_dest_filter "$NAME" "$THEFILTER" break else TMPFILTER=`mk_filter_rules "$PART"` if [ -n "$THEFILTER" ] then THEFILTER="$THEFILTER or $TMPFILTER" else THEFILTER="$TMPFILTER" fi fi done else NAME=`basename $DEST|sed 's/\./_/g'` TMPFILTER=`mk_filter_rules "$LINE"` mk_dest_filter "$NAME" "$TMPFILTER" fi } proceed() { LINE="$1" RULES=`echo "$LINE"|cut -d" " -f1` DEST=`echo "$LINE"|cut -d" " -f2-10|sed 's/^-//g'` if [ "$DEST" = "*" ] then DEST="/var/log/everything.log" fi proceedline "$RULES" "$DEST" WHAT=`basename $DEST|sed 's/\./_/g'` mk_log_entry "f_$WHAT" "d_$WHAT" $LINE } ############################################# # main program ############################################# # statical entrys header echo mk_log_src echo mk_internal_entry echo # dynamical entries cat "$INFILE" |grep -v "^#"|grep -v "^$" |expand|sed 's/ */ /'| \ while read LINE do proceed "$LINE" done

1 0

RE: [syslog-ng] syslog-ng GUI
by Dukes Clayton 11 May '06

11 May '06

Uh, ok... So can someone dumb those pdf's down so that someone with only a 150iq can understand them? Geeze man. I'm working on a development version of php-syslog-ng that does (I think) what sisyphus is referring to. Essentially, group multiple, repeated tokens into the DB thereby decreasing large amounts of data. When a token gets repeated, simply update a count field using "ON DUPLICATE KEY UPDATE count = count + 1" (available in MySQl v4.1+) I just need the time to re-write the tables. I've already done it with the hosts table: I created a new table called hosts and just update the count field, then reference that host using a foreign key in the logs table. -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Jon Stearley Sent: Thursday, May 11, 2006 10:50 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] syslog-ng GUI On May 11, 2006, at 6:51 AM, Arya, Manish Kumar wrote: > Hi Guys, > > I am storing logs on a central server having 3T SAN, using follwing > template > > destination indexlog { > > file("/logs/log01/indexlog/$YEAR/$MONTH/$DAY/$HOST" > > template("$HOUR:$MIN:$SEC,$PROGRAM,$FACILITY,$PRIORITY,$MSGONLY\n") > template-escape(yes) > owner(root) group(root) perm(0644) > dir_perm(0755) create_dirs(yes)); > }; > > my logging is done perfectly :) > > like /logs/log01/indexlog/2006/05/11/hostnames > > I want to have a GUI to view logs with following facilities > > -search logs on basis on date/time, text patterns in > messages,hostnames. http://www.cs.sandia.gov/sisyphus/ mines patterns, but does not have a production GUI (yet). It is more of a research tool at this point, but I would be happy to help you give it a try. Recent emphasis has been on the functionality described in .../detection.pdf. Please let me know if interested, like I said I'd be happy to help, and am in fact looking for additional datasets to analyze; I find my approach to be effective for supercomputer logs, but have not yet explored its effectiveness for other log sets (eg enterprise). I've been waiting to implement a production GUI until I am confident that the underlying functionality is general and excellent. My current leaning is towards adding sisyphus functionality to splunk's interface (and have contacted splunk about this). G'day! -- +--------------------------------------------------------------+ | Jon Stearley (505) 845-7571 (FAX 844-9297) | | Sandia National Laboratories Scalable Systems Integration | +--------------------------------------------------------------+ _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

1 0

RE: [syslog-ng] syslog-ng GUI
by Dukes Clayton 11 May '06

11 May '06

Try Php-syslog-ng -----Original Message----- From: syslog-ng-bounces(a)lists.balabit.hu [mailto:syslog-ng-bounces@lists.balabit.hu] On Behalf Of Arya, Manish Kumar Sent: Thursday, May 11, 2006 7:51 AM To: syslog-ng(a)lists.balabit.hu Subject: [syslog-ng] syslog-ng GUI Hi Guys, I am storing logs on a central server having 3T SAN, using follwing template destination indexlog { file("/logs/log01/indexlog/$YEAR/$MONTH/$DAY/$HOST" template("$HOUR:$MIN:$SEC,$PROGRAM,$FACILITY,$PRIORITY,$MSGONLY\n") template-escape(yes) owner(root) group(root) perm(0644) dir_perm(0755) create_dirs(yes)); }; my logging is done perfectly :) like /logs/log01/indexlog/2006/05/11/hostnames I want to have a GUI to view logs with following facilities -search logs on basis on date/time, text patterns in messages,hostnames. -should provide facility to have filters associated with user authentication/authorization. -should be able to to parallel search to improve search response time. Regards, -Manish __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com _______________________________________________ syslog-ng maillist - syslog-ng(a)lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html

2 1

Multi-threading Syslog-ng
by jarrod.schous＠sasktel.sk.ca 10 May '06

10 May '06

Is syslog-ng capable of multi-threading?. I.E. Can it distribute its processing requirements accross multiple CPU's. I have seen a couple side notes that say it isn't but I was wondering if there is a definitive answer. Thanks Jarrod. NOTICE: This confidential e-mail message is only for the intended recipient(s). If you are not the intended recipient, be advised that disclosing, copying, distributing, or any other use of this message, is strictly prohibited. In such case, please destroy this message and notify the sender.

2 1

cnnection 0 desn't have any source
by wayne yu 10 May '06

10 May '06

Hi everyone: I got waning message when I starting my syslog : Warning: connection 0 doesn't have any sources and I don't how to fixing it. can someone look at my /etc/syslog-ng.conf and tell me how to fixing it. # Options options { use_fqdn(yes); sync(0); keep_hostname(yes); chain_hostnames(no); create_dirs(yes); }; # Sources of syslog messages (both local and remote messages on the server) source s_local { sun-streams("/dev/log" door("/etc/.syslog_door")); internal(); }; source s_stunnel { tcp(ip("127.0.0.1") port(514) max-connections(1)); }; source s_udp { udp(); }; # Level Filters filter f_emerg { level (emerg); }; filter f_alert { level (alert .. emerg); }; filter f_crit { level (crit .. emerg); }; filter f_err { level (err .. emerg); }; filter f_warning { level (warning .. emerg); }; filter f_notice { level (notice .. emerg); }; filter f_info { level (info .. emerg); }; filter f_debug { level (debug .. emerg); }; # Facility Filters filter f_kern { facility (kern); }; filter f_user { facility (user); }; filter f_mail { facility (mail); }; filter f_daemon { facility (daemon); }; filter f_auth { facility (auth); }; filter f_syslog { facility (syslog); }; filter f_lpr { facility (lpr); }; filter f_news { facility (news); }; filter f_uucp { facility (uucp); }; filter f_cron { facility (cron); }; filter f_local0 { facility (local0); }; filter f_local1 { facility (local1); }; filter f_local2 { facility (local2); }; filter f_local3 { facility (local3); }; filter f_local4 { facility (local4); }; filter f_local5 { facility (local5); }; filter f_local6 { facility (local6); }; filter f_local7 { facility (local7); }; # Custom Filters filter f_user_none { not facility (user); }; filter f_kern_debug { filter (f_kern) and filter (f_debug); }; filter f_daemon_notice { filter (f_daemon) and filter (f_notice); }; filter f_mail_crit { filter (f_mail) and filter (f_crit); }; filter f_mesg { filter (f_kern_debug) or filter (f_daemon_notice) or filter (f_mail_crit); }; filter f_authinfo { filter (f_auth) or program (sudo); }; destination hosts { file("/var/log/HOSTS/$HOST/$YEAR/$MONTH/$DAY/$FACILITY$YEAR$MONTH$DAY" owner(root) group(root) perm(0600) dir_perm(0700) create_dirs(yes)); }; #Destinations: local files, the console, and the client files destination l_authlog { file ("/var/log/authlog"); }; destination l_messages { file ("/var/log/messages"); }; destination l_maillog { file ("/var/log/maillog"); }; destination l_ipflog { file ("/var/log/ipflog"); }; destination l_imaplog { file ("/var/log/imaplog"); }; destination l_syslog { file ("/var/log/syslog"); }; destination l_console { file ("/dev/console"); }; destination r_authlog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/authlog"); }; destination r_messages { file ("/var/log/clients/$YEAR/$MONTH/$HOST/messages"); }; destination r_maillog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/maillog"); }; destination r_ipflog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/ipflog"); }; destination r_imaplog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/imaplog"); }; destination r_console { file ("/var/log/clients/$YEAR/$MONTH/$HOST/consolelog"); }; destination r_syslog { file ("/var/log/clients/$YEAR/$MONTH/$HOST/syslog"); }; destination r_fallback { file ("/var/log/clients/$YEAR/$MONTH/$HOST/$FACILITY-$LEVEL"); }; destination mail-alert-perl { program("/usr/local/bin/syslog-mail-perl"); }; filter f_attack_alert { match("attackalert"); }; # find messages with "attackalert" in them, and send to the mail-alert script log { #source(src); filter(f_attack_alert); destination(mail-alert-perl); }; # Log statements # Local sources log { source (s_local); filter (f_authinfo); destination (l_authlog); }; log { source (s_local); filter (f_mail); destination (l_maillog); }; log { source (s_local); filter (f_local0); destination (l_ipflog); }; log { source (s_local); filter (f_local1); destination (l_imaplog); }; log { source (s_local); filter (f_syslog); destination (l_syslog); }; log { source (s_local); filter (f_emerg); filter (f_user_none); destination (l_console); }; log { source (s_local); filter (f_mesg); filter (f_user_none); destination (l_messages); }; # All sources, since we want to archive local and remote logs log { source (s_local); source (s_stunnel); filter (f_authinfo); destination (r_authlog); }; log { source (s_local); source (s_stunnel); filter (f_mail); destination (r_maillog); }; log { source (s_local); source (s_stunnel); filter (f_local0); destination (r_ipflog); }; log { source (s_local); source (s_stunnel); filter (f_local1); destination (r_imaplog); }; log { source (s_local); source (s_stunnel); filter (f_syslog); destination (r_syslog); }; log { source (s_local); source (s_stunnel); filter (f_emerg); filter (f_user_none); destination (l_console); }; log { source (s_local); source (s_stunnel); filter (f_mesg); filter (f_user_none); destination (l_messages); }; Thanks in advance. Wayne Yu __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

2 1