September 2010 - syslog-ng - lists.balabit.hu

Rewrite Rule Syntax Confusion
by Matthew Hall 02 Sep '10

02 Sep '10

Hello all, I am trying to make a combination of a template and a rewrite rule to write my messages into an extension of WELF format to make them easier to process downstream (the entire point of patterndb I guess). The template part was trivial and works perfectly. I just used some regex hacking to extract all 45 variable names I used in my patterndbs, then used gawk to dump them out into a giant (escaped) string like this, and pasted it to the config file. template("var1=\"$var1\" var2=\"$var2\" ...\n"); One minor thing: Currently I am forced to expand varfoo="$varfoo" even if the value of $varfoo is empty because a particular message did not contain "$varfoo". I suspect I could fix this by writing a C plugin which checks the length before expanding but I wondered if there were an easier way. Else maybe we could make this WELF output techniquie available for everybody because I think the problem could be very common. In any case I can always make the downstream ignore empty expansions for now. The only problem: I want one field at the end of the message to be named 'raw' and contain a double-quote-escaped copy of the raw message. That way if the message contains double quote characters it won't break a parser because they will be escaped right. Here is how I thought of doing this after reading the manual: rewrite r_add_raw { set(subst("\"", "\\\"", value("$MSGHDR$MSG"), flags("global")), value("raw")); } However I am getting hit with this syntax error: WARNING: Your configuration uses a newly introduced reserved word as identifier, please use a different name; keyword='subst', filename='welf.conf', line='2' Apparently I did it wrong, or there is a reason I can't call set on my raw variable to set the raw variable to the double quote escaped string subst'ed from input message in $MSGHDR$MSG. I really hope somebody can comment about how to fix it so that double quotes in the input won't blow things up. For now I worked around it temporarily just to get up and running by adding raw=\"$MSGHDR$MSG\" to the end of my template just to see if it would work OK and it does work OK when you don't have double quotes in there. Matthew.

2 1

Latest and greatest PatternDBs
by Matthew Hall 02 Sep '10

02 Sep '10

Hello all, Thanks to the great help you guys provided earlier I have gotten the hang of patterndb. I am still working on whether I can open-source mine or not, I hope my employer will allow it. I am wondering if this file is the latest and greatest? It's sort of old. http://www.balabit.com/downloads/files/patterndb-snapshot/patterndb-2009120… Thanks, Matthew.

2 1

configure installation path in a on-interactive installation
by Hoenig, Stefan, VF-Group 02 Sep '10

02 Sep '10

Hi there, is it possible to install syslog-ng PE to another path than the default /opt/syslog-ng and use this argument in a non-interactive installation. I tried to configure it by using the .run installer but no option is displayed to change the path. Any ideas or commentrs ??? With best regards / Mit freundlichen Grüßen Stefan Hoenig

2 1

syslog-ng 3.1.2-1 problem/bug/?
by VDR User 02 Sep '10

02 Sep '10

Hi. I'm using Debian testing and "upgraded" from 3.1.1-3 to 3.1.2-1 earlier. Now I'm getting "Error opening file for writing; filename='/logs/vdr.log', error='Permission denied (13)'".. This problem does _not_ occur with 3.1.1-3, which I have downgraded back to (until the problem/bug is solved). Are there any known issues causing this? I don't believe there are any permission problems considering the user owns that dir and it works fine in 3.1.1-3. Following is the diff between the syslog-ng.conf's. The - lines are from my working 3.1.1-3 conf, the + lines are from the 3.1.2-1 conf.... Any help is appreciated. Best regards, Derek --- /etc/syslog-ng/syslog-ng.conf 2010-08-15 12:01:35.000000000 -0700 +++ /etc/syslog-ng/syslog-ng.conf.dpkg-new 2010-08-04 12:07:22.000000000 -0700 @@ -6,7 +6,8 @@ # First, set some global options. options { long_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); - owner("root"); group("adm"); perm(0640); + owner("root"); group("adm"); perm(0640); stats_freq(0); + bad_hostname("^gconfd$"); }; ######################## @@ -32,10 +33,10 @@ destination d_auth { file("/var/log/auth.log"); }; destination d_cron { file("/var/log/cron.log"); }; destination d_daemon { file("/var/log/daemon.log"); }; -destination d_kern { file("/var/log/kern.log" template("$FULLDATE $MESSAGE\n")); }; +destination d_kern { file("/var/log/kern.log"); }; destination d_lpr { file("/var/log/lpr.log"); }; destination d_mail { file("/var/log/mail.log"); }; -destination d_syslog { file("/var/log/syslog" template("$FULLDATE $MESSAGE\n")); }; +destination d_syslog { file("/var/log/syslog"); }; destination d_user { file("/var/log/user.log"); }; destination d_uucp { file("/var/log/uucp.log"); }; @@ -106,7 +107,7 @@ local6, local7) and not filter(f_debug); }; filter f_mail { facility(mail) and not filter(f_debug); }; filter f_news { facility(news) and not filter(f_debug); }; -filter f_syslog3 { facility(syslog) and not filter(f_debug); }; +filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; filter f_user { facility(user) and not filter(f_debug); }; filter f_uucp { facility(uucp) and not filter(f_debug); }; @@ -152,14 +153,3 @@ # All messages send to a remote site # #log { source(s_src); destination(d_net); }; - - -######################## -# VDR LOGGING -######################## -destination d_vdr { file("/logs/vdr.log" template("$FULLDATE $MESSAGE\n") owner("test") group("staff") perm(0640)); }; - -filter f_vdr { program("vdr"); }; -filter f_novdr { not program("vdr"); }; - -log { source(s_src); filter(f_vdr); destination(d_vdr); };

2 3

changing incoming facility.level -> kern.emerg
by Charlie Reddington 02 Sep '10

02 Sep '10

I'm having some funky behavior where my central syslog-ng server is marking all forwarded events as facility kern, level emerg. The remote hosts, are CentOS 5 x86_64. They are using the system install syslog. The ones I started with, have nginx web server, patched to send it's log messages under local5.crit for errors, and local5.notice for access logs. It's syslog.conf looks like so. Code: # cat /etc/syslog.conf # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # nginx local logs local5.notice /usr/ local/logs/access.log local5.crit /usr/ local/logs/error.log # Syslog-ng *.* @remoteIP They are logging locally as expected, and things are coming into the remote host as expected. (from a strace) 32443 recvfrom(3, "<173>nginx: ScrubbedIP www.fa"..., 8192, 0, {sa_family=AF_INET, sin_port=htons(514), sin_addr=inet_addr("ScrubbedIP")}, [16]) = 412 What I can't tell is how they are actually coming in (meaning what level / facility). My central logging server is running the latest syslog-ng. My configs look like... Code: # cat /etc/syslog-ng/syslog-ng.conf # Syslog-ng config options { use_dns(no); keep_hostname(yes); long_hostnames(off); chain_hostnames(off); sync(1); log_fifo_size(1024); create_dirs(yes); perm(0644); stats_freq(3600); mark_freq(600); }; # # Data Sources # # External Syslog source s_external { # udp(); udp(ip("0.0.0.0") port(514) flags(no_parse)); }; # # Where to put Data # # Access Logs destination d_access_log { file("/syslog-ng/$HOST/access.log"); }; # Error Logs destination d_error_log { file("/syslog-ng/$HOST/error.log"); }; # Everything else destination d_log { file("/syslog-ng/$HOST/$FACILITY.$LEVEL"); }; # # Data Filters # # Facility local5 filter f_local5 { facility(local5); }; # Level info filter f_info { level(info); }; # Level notice filter f_notice { level(notice); }; # Level warn filter f_warn { level(warn); }; # Level crit filter f_crit { level(crit); }; # Level err filter f_err { level(err); }; # # Data Logging Locations # # Access Log log { source(s_external); filter(f_local5); filter(f_notice); destination(d_access_log); }; # Error Log log { source(s_external); filter(f_local5); filter(f_crit); destination(d_error_log); }; log { source(s_external); destination(d_log); }; The only problem here is that I'm not getting any access.log or error.log. I'm only getting kern.emerg log with EVERYTHING in it (the last log rule). I have verified that events are coming in correctly as I expect. Code: tcpdump port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 15:47:33.128579 IP scrubbed.syslog > scrubbed.syslog: SYSLOG local5.notice, length: 409 15:47:33.145029 IP scrubbed.syslog > scrubbed.syslog: SYSLOG local5.notice, length: 233 So I'm not sure what syslog-ng is doing to convert this to kern.emerg instead of leaving it local5.notice. At this point, I'm pretty sure the config is sane, but maybe I'm missing something. Syslog-NG / central server stats # /sbin/syslog-ng -V syslog-ng 2.1.4 # cat /etc/redhat-release CentOS release 5.5 (Final) Fetched with GIT. syslog / remote server stats # /sbin/syslogd -v syslogd 1.4.1 # cat /etc/redhat-release CentOS release 5.5 (Final) Any help is appreciated. Charlie

2 1