- syslog-ng - lists.balabit.hu

Insert hostname instead of overwrite ?
by stucky 03 Sep '10

03 Sep '10

Guys I'm trying to log to a loadbalanced VIP. It seems to work ok except that the loadbalancer uses SNAT so I loose my source IP. This means I cannot use dns or even the source ip to get the source host as all logs appear to come from the same source (the loadbalancer). This means I have no choice but to rely on the hostname field which works about 98% of the time but some stuff like Dell OpenManage skips the hostname field. So I'd get logs like this on host "cage" f.e. Aug 16 21:47:22 Server Administrator: Storage Service EventID: 2242 The Patrol Read has started.: Controller 0 (PERC 5/i Integrated) I fixed that by telling the syslog-ng client to force itself to figure out a proper hostname and now the log looks like this Aug 17 13:51:10 cage Administrator[]: Instrumentation Service EventID: 1000 Server Administrator starting I thought syslog-ng inserts the hostname but by the looks of it it simply replaces whatever is in the expected field with the hostname it has just figured out. As you can see it overwrote the entry "Server". No biggie in the above case but what if this field contained valuable information ? I'd loose that. Any way to squeeze in the hostname so to speak ? -- stucky

5 12

[PATCH] remove auto-generated files
by chris packham 03 Sep '10

03 Sep '10

INSTALL is generated by the build process. If the build host has a different version of autotools then the contents of INSTALL can change which in turn causes problems for developers working from the git repository. This patch removes the INSTALL file and adds it to .gitignore. --- Hi Bazsi, This has been causing us a little bit of a hassle with rebasing our syslog-ng clone against the upstream one. Hopefully it doesn't cause problems for your build processes. .gitignore | 1 + INSTALL | 182 ------------------------------------------------------------ 2 files changed, 1 insertions(+), 182 deletions(-) delete mode 100644 INSTALL diff --git a/.gitignore b/.gitignore index b2bfbec..b5e59f6 100644 --- a/.gitignore +++ b/.gitignore @@ -20,3 +20,4 @@ stamp-h ylwrap .*.swp *.pyc +INSTALL diff --git a/INSTALL b/INSTALL deleted file mode 100644 index 47bbab8..0000000 --- a/INSTALL +++ /dev/null @@ -1,182 +0,0 @@ - -Installation instructions for SYSLOG-NG -======================================= - -Welcome. This is syslog-ng, which stands for syslog-new-generation, a new, -enhanced system logging daemon. - -Unpacking the distribution -========================== - -The distribution arrives in .tar.gz format though OS/distribution packaging -is possible. The file is named: - - syslog-ng-x.xx.tar.gz - -where x.xx stands for the version number. You must have tar and gzip to -unpack the distribution (sorry, compress is not supported). If you have GNU -tar simply execute the following command: - - tar xvfz syslog-ng-x.xx.tar.gz - -If your version of tar doesn't support z (most non-GNU tars), you should -execute this one: - - gunzip -c syslog-ng-x.xx.tar.gz | tar xvf - - -After this, you'll get a directory named syslog-ng-x.xx, where the source for -syslog-ng will be unpacked. - -Compiling the program: -====================== - -syslog-ng requires gcc as a C compiler (at least version 2.7.2), GNU flex as -a lex, and bison as a parser generator. Some GNU C and GNU flex extensions -are used, porting to other compilers/lex/yacc combination is welcome. - -Syslog-ng relies on a number of third party libraries in order to be -compiled, these are: - * libnet if spoof-source support is enabled in configure - (--enable-spoof-source) - * libwrap (aka TCP wrappers) if their use is enabled in configure - (--enable-tcp-wrapper) - * Glib a generic I/O library used by a wide variety of applications - * EventLog a generic event logging library developed by BalaBit - -Once these libraries are installed, you can start compiling syslog-ng: - -cd to the syslog-ng-x.xx directory, and execute the following commands: - - ./configure - make - -After the make cycle finishes, you'll get an executable in the src -directory: - - syslog-ng - the main binary - -Now do a "make install" and you are done. - -Compile time options -==================== -A couple of features must be enabled in compile time using one of the ---enable-<feature> option to the configure script. - -Currently the following compile time features exist: - - --enable-debug include debug info - --enable-sun-streams enable Sun STREAMS support even if not detected (autodetected by default) - --enable-sun-door enable Sun door support even if not detected (autodetected by default) - --enable-tcp-wrapper enable using /etc/hosts.deny & /etc/hosts.allow for TCP access (disabled by default) - --enable-spoof-source enable spoof_source feature (also requires libnet) (disabled by default) - --enable-ipv6 enable support for IPv6 - --enable-static-linking compile syslog-ng as a static binary - --disable-dynamic-linking compile syslog-ng against system libraries - dynamically, and statically against everything else - -NOTE: if you intend to use an NFS mounted /usr you might need to configure -syslog-ng with either --enable-static-linking or with ---disable-dynamic-linking as syslog-ng needs to start _before_ your -networking is up. - -Platform specific compilation issues -==================================== - - HP-UX with gcc 4.x - ------------------ - - The gcc available for HP-UX gives an error message in sys/socket.h, as a - function is defined as static and extern at the same time. Earlier gcc - versions and the HP compiler only gives a warning, gcc4 issues an error. - - The solution is to copy sys/socket.h to your gcc-private include - directory and fix the offending declaration. - -Configuration file: -=================== - -Syslog-ng uses a different configuration scheme than the original syslogd, -which sits at /etc/syslog-ng/syslog-ng.conf (or at a different location -depending the value passed to the --sysconfdir configure parameter). - -The manpage for syslog-ng.conf(5) or the documentation under doc contains a -/reference about keywords and syntax which can be used in the -config file. For now I only explain system dependencies. - -Every unix version has a slightly different way of sending log messages, and -since syslog-ng gives you the power of choosing your log-sources, you have to -be aware some of the internals. - - Linux: - ------ - Linux has a dedicated unix socket called /dev/log, where log messages are - written to, and read from. It is of type SOCK_STREAM. So the correct source - statement for standard linux log messages is: - - source stdlog { unix-stream("/dev/log"); }; - - Some newer Linux distributions (Debian GNU/Linux woody, and RedHat Linux - post 7.0) switched over to using SOCK_DGRAM in their sysklogd installation - by default. The libc autodetects which socket type is in use, but some - programs (like klogd) log directly to the log device bypassing libc. So if - you have logging problems you might want to switch log unix-dgram like - this: - - source stdlog { unix-dgram("/dev/log"); }; - - BSDi: - ----- - BSD is similar to Linux (or vice-versa Linux is similar to BSD, but this is - another issue), so BSD has also a unix socket for log communication, but - it's of type SOCK_DGRAM, and it is located at /var/run/log. So the source - statement you are looking for is: - - source stdlog { unix-dgram("/var/run/log"); }; - - Solaris (2.5.1 or below): - ------------------------- - - SunOS/Solaris has a universal means of communications called STREAMS. It is - used as both an in-kernel and kernel-user interface. You'll need to feed the - following statement to syslog-ng to accept all messages: - - source stdlog { sun-stream("/dev/log"); }; - - Solaris (2.6 - Solaris8) - ------------------- - - In addition to the STREAMS device used in earlier versions, a door is used to - make sure after each message that the system logging daemon is still running. - To create that door, you'll need the door() option of the sun-stream driver: - Sun has added a new method to the pool of possible IPC mechanisms, and it - is called door. syslog-ng supports this method with the sun-door keyword. A - door is a special file in the filesystem, and is called /etc/.syslog_door. - So your correct source statement would be: - - source stdlog { sun-streams("/dev/log" door("/etc/.syslog_door")); }; - - Solaris 9 - --------- - - The name of the door file has been changed from /etc/.syslog_door to - /var/run/syslog_door, the correct configuration needs to be: - - source stdlog { sun-streams("/dev/log" door("/var/run/syslog_door")); }; - - AIX (unknown revision) - ---------------------- - - AIX does support STREAMS, but its log transport doesn't use it. As it - seems /dev/log is a simple SOCK_DGRAM type unix socket, so it works using: - - source stdlog { unix-dgram("/dev/log"); }; - - HP-UX (HP-UX 11.0) - ------------------ - - HP-UX uses a named pipe called /dev/log for log transport, and you - can use this with the pipe() driver with an additional option. HP-UX pads - all incoming messages to 2048 bytes by default, so you need to specify - this: - - source stdlog { pipe("/dev/log" pad_size(2048)); }; -- 1.7.2.2

2 1

SNMP Traps?
by Clayton Dukes 03 Sep '10

03 Sep '10

Hey guys, A while back, I though I read that syslog-ng would process traps as well in v3.x, was I just imagining that or did it happen? ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 3

TCP recv bug in syslog-ng v2.09?
by Clayton Dukes 02 Sep '10

02 Sep '10

Hey guys, Are there any known bugs for syslog-ng v2.09 that won't allow a cisco router to send logs over tcp? I can see a connection established in syslog-ng. I also see the message come in via tcpdump, but nothing in syslog-ng's output. If I change the router from tcp to udp, messages come in as expected. *Router config:* logging source-interface Loopback0 logging 172.18.224.150 <tricon:40,-1|172.18.224.150;majoshi@cisco.com> logging host 172.18.224.190 <tricon:40,-1|172.18.224.190;majoshi@cisco.com>transport tcp *syslog-ng config:* source s_all { udp(); tcp(ip(11.31.130.99) port(8002) max-connections(300)); tcp(ip(172.18.224.190) port(601) max-connections(300)); }; *debug output:* I commented out the line above for the other interface (11.31.130.99), restarted and this is all I see: Syslog connection accepted; from='AF_INET(14.3.23.50<tricon:40,-1|(14.3.23.50;majoshi@cisco.com>:63845)', to='AF_INET(172.18.224.190 <tricon:40,-1|(172.18.224.190;majoshi@cisco.com> :601)' *tcpdump:* 14:13:46.914566 IP (tos 0x0, ttl 251, id 4303, offset 0, flags [none], proto TCP (6), length 134) 14.3.23.50.63845 > xxx.com.601: Flags [.], seq 230:324, ack 1, win 4128, length 94 *Router debug:* *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:19.772<tricon:40,-1|.772;majoshi@cisco.com>: %SYS-5- <tricon:40,-1|-5-;majoshi@cisco.com>CONFIG_I: Configured from console by pnoc on vty0 (172.18.224.151) <tricon:40,-1|0 (172.18.224.151);majoshi(a)cisco.com> *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:20.776<tricon:40,-1|.776;majoshi@cisco.com>: Released port 15205 <tricon:40,-1|15205;majoshi@cisco.com> in Transport Port Agent for TCP IP type 1 delay 240000 <tricon:40,-1|240000;majoshi@cisco.com> *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:20.776<tricon:40,-1|.776;majoshi@cisco.com>: TCB 0x850 <tricon:40,-1|850;majoshi@cisco.com>F9754<tricon:40,-1|9754;majoshi@cisco.com>destroyed *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: TCB83648 <tricon:40,-1|83648;majoshi@cisco.com>E60 created *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: TCB83648 <tricon:40,-1|83648;majoshi@cisco.com>E60 setting property TCP_PID (8) 845083 <tricon:40,-1|(8) 845083;majoshi(a)cisco.com>E4 *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: TCB83648 <tricon:40,-1|83648;majoshi@cisco.com>E60 setting property TCP_NO_DELAY (1) 845083 <tricon:40,-1|(1) 845083;majoshi(a)cisco.com>E8 *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: TCB83648 <tricon:40,-1|83648;majoshi@cisco.com>E60 setting property TCP keepalive timeout (17) 845084 <tricon:40,-1|(17) 845084;majoshi(a)cisco.com> A0 *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: TCP: Random local port generated 63845<tricon:40,-1|63845;majoshi@cisco.com>, network 1 *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: TCB83648 <tricon:40,-1|83648;majoshi@cisco.com>E60 bound to 14.3.23.50.63845<tricon:40,-1|14.3.23.50.63845;majoshi@cisco.com> *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: Reserved port 63845 <tricon:40,-1|63845;majoshi@cisco.com> in Transport Port Agent for TCP IP type 1 *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: TCP: sending SYN, seq 3300233565 <tricon:40,-1|3300233565;majoshi@cisco.com>, ack 0 *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: TCP0: Connection to 172.18.224.190<tricon:40,-1|172.18.224.190;majoshi@cisco.com>:601, advertising MSS 536 <tricon:40,-1|536;majoshi@cisco.com> *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.775<tricon:40,-1|.775;majoshi@cisco.com>: TCP0: state was CLOSED -> SYNSENT [63845 - <tricon:40,-1|63845 -;majoshi(a)cisco.com>> 172.18.224.190(601)<tricon:40,-1|172.18.224.190(601);majoshi@cisco.com> ] *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.779<tricon:40,-1|.779;majoshi@cisco.com>: TCP0: state was SYNSENT -> ESTAB [63845 - <tricon:40,-1|63845 -;majoshi(a)cisco.com>> 172.18.224.190(601)<tricon:40,-1|172.18.224.190(601);majoshi@cisco.com> ] *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.779<tricon:40,-1|.779;majoshi@cisco.com>: TCP: tcb 83648 <tricon:40,-1|83648;majoshi@cisco.com>E60 connection to 172.18.224.190 <tricon:40,-1|172.18.224.190;majoshi@cisco.com>:601, peer MSS 1460 <tricon:40,-1|1460;majoshi@cisco.com>, MSS is 536<tricon:40,-1|536;majoshi@cisco.com> *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.779<tricon:40,-1|.779;majoshi@cisco.com>: TCB83648 <tricon:40,-1|83648;majoshi@cisco.com>E60 connected to 172.18.224.190.601 <tricon:40,-1|172.18.224.190.601;majoshi@cisco.com> *Aug 17 17 <tricon:40,-1|17 17;majoshi(a)cisco.com>:34:25.779<tricon:40,-1|.779;majoshi@cisco.com>: %SYS-6- <tricon:40,-1|-6-;majoshi@cisco.com>LOGGINGHOST_STARTSTOP: Logging to host 172.18.224.190 <tricon:40,-1|172.18.224.190;majoshi@cisco.com> port 601 <tricon:40,-1|601;majoshi@cisco.com> started - reconnection ______________________________________________________________ Clayton Dukes ______________________________________________________________

5 15

v3.0.8: How to logging messages with non-rfc timestamps?
by Sergey V. Lobanov 02 Sep '10

02 Sep '10

I'm using syslog-ng v3.0.8(OSE) for logging messages from remote devices. There is a host which sends messages with incorrect timestamp format, for example "Jan 19 15:52:7". Syslog-ng does not store the first word in timestamp via macros $MSGHDR. For example: $ echo "<134>Jan 19 15:52:7 test" | nc -u 192.168.3.142 514 ^C But log file does not contains the word 'Jan': Aug 29 16:49:10 10.147.0.202 10.147.0.202 19 15:52:7 test Configuration file: @version: 3.0 options { use_dns(yes); }; source src_remote { udp(ip("0.0.0.0") port(514) flags(store-legacy-msghdr)); tcp(ip("0.0.0.0") port(514) flags(store-legacy-msghdr)); }; template t_remote {template("$DATE $SOURCEIP $HOST_FROM $MSGHDR$MESSAGE\n"); template_escape(no); }; destination remoteall { file("/var/log/all-remote.log" template(t_remote)); }; log { source(src_remote); destination(remoteall); }; How can I store the original timestamp from the remote host? -- wbr, Sergey V. Lobanov

2 2

Syslog-ng on OpenBSD
by Joe McDonagh 02 Sep '10

02 Sep '10

Nothing is being logged except Log Processing Statistics in /var/log/debug. I am seeing a ton of dropped stuff to my loghost, but what's really alarming is that stuff isn't being logged locally? Sep 2 11:10:02 machine syslog-ng[4179]: Log statistics; dropped='tcp(localhost:61514)=48833', processed='center(queued)=52', processed='center(received)=13', processed='destination(messages)=13', processed='destination(mailinfo)=0', processed='destination(mailwarn)=0', processed='destination(console)=0', processed='destination(mailerr)=0', processed='destination(loghost)=13', processed='destination(authlog)=0', processed='destination(user)=0', processed='destination(lpr)=0', processed='destination(daemon)=0', processed='destination(cron)=0', processed='destination(kern)=0', processed='destination(mail)=0', processed='destination(debug)=13', processed='destination(syslog)=13', processed='destination(console_all)=0', processed='source(src)=13', suppressed='tcp(localhost:61514)=0' This is happening on all my OpenBSD machines I recently setup to use syslog-ng (needed TCP with stunnel in front to send to my log host). This is the config: # This file is managed by puppet as a file in the syslog::client class options { long_hostnames(off); sync(20); use_dns(no);}; source src { internal(); pipe("/dev/klog" log_prefix("kernel: ")); unix-stream("/dev/log"); }; destination authlog { file("/var/log/auth.log"); }; destination syslog { file("/var/log/syslog"); }; destination cron { file("/var/log/cron.log"); }; destination daemon { file("/var/log/daemon.log"); }; destination kern { file("/var/log/kern.log"); }; destination lpr { file("/var/log/lpr.log"); }; destination user { file("/var/log/user.log"); }; destination mail { file("/var/log/mail.log"); }; destination mailinfo { file("/var/log/mail.info"); }; destination mailwarn { file("/var/log/mail.warn"); }; destination mailerr { file("/var/log/mail.err"); }; destination debug { file("/var/log/debug"); }; destination messages { file("/var/log/messages"); }; destination console { usertty("root"); }; destination console_all { file("/dev/tty12"); }; destination loghost { tcp("localhost" port(61514)); }; filter f_auth { facility(auth); }; filter f_authpriv { facility(auth, authpriv); }; filter f_syslog { not facility(authpriv, mail); }; filter f_cron { facility(cron); }; filter f_daemon { facility(daemon); }; filter f_kern { facility(kern); }; filter f_lpr { facility(lpr); }; filter f_mail { facility(mail); }; filter f_user { facility(user); }; filter f_debug { not facility(auth, authpriv, mail); }; filter f_messages { level(info..warn) and not facility(auth, authpriv, mail); }; filter f_emergency { level(emerg); }; filter f_info { level(info); }; filter f_notice { level(notice); }; filter f_warn { level(warn); }; filter f_crit { level(crit); }; filter f_err { level(err); }; log { source(src); filter(f_authpriv); destination(authlog); }; log { source(src); filter(f_syslog); destination(syslog); }; log { source(src); filter(f_cron); destination(cron); }; log { source(src); filter(f_daemon); destination(daemon); }; log { source(src); filter(f_kern); destination(kern); }; log { source(src); filter(f_lpr); destination(lpr); }; log { source(src); filter(f_mail); destination(mail); }; log { source(src); filter(f_user); destination(user); }; log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); }; log { source(src); filter(f_mail); filter(f_warn); destination(mailwarn); }; log { source(src); filter(f_mail); filter(f_err); destination(mailerr); }; log { source(src); filter(f_debug); destination(debug); }; log { source(src); filter(f_messages); destination(messages); }; log { source(src); filter(f_emergency); destination(console); }; log { source(src); destination(loghost); }; <<EOF>> I'm wondering if my source line is just wrong for OpenBSD. Anybody with a working OpenBSD config care to share? -- Joe McDonagh AIM: YoosingYoonickz IRC: joe-mac on freenode "When the going gets weird, the weird turn pro."

2 2

Rewrite Rule Syntax Confusion
by Matthew Hall 02 Sep '10

02 Sep '10

Hello all, I am trying to make a combination of a template and a rewrite rule to write my messages into an extension of WELF format to make them easier to process downstream (the entire point of patterndb I guess). The template part was trivial and works perfectly. I just used some regex hacking to extract all 45 variable names I used in my patterndbs, then used gawk to dump them out into a giant (escaped) string like this, and pasted it to the config file. template("var1=\"$var1\" var2=\"$var2\" ...\n"); One minor thing: Currently I am forced to expand varfoo="$varfoo" even if the value of $varfoo is empty because a particular message did not contain "$varfoo". I suspect I could fix this by writing a C plugin which checks the length before expanding but I wondered if there were an easier way. Else maybe we could make this WELF output techniquie available for everybody because I think the problem could be very common. In any case I can always make the downstream ignore empty expansions for now. The only problem: I want one field at the end of the message to be named 'raw' and contain a double-quote-escaped copy of the raw message. That way if the message contains double quote characters it won't break a parser because they will be escaped right. Here is how I thought of doing this after reading the manual: rewrite r_add_raw { set(subst("\"", "\\\"", value("$MSGHDR$MSG"), flags("global")), value("raw")); } However I am getting hit with this syntax error: WARNING: Your configuration uses a newly introduced reserved word as identifier, please use a different name; keyword='subst', filename='welf.conf', line='2' Apparently I did it wrong, or there is a reason I can't call set on my raw variable to set the raw variable to the double quote escaped string subst'ed from input message in $MSGHDR$MSG. I really hope somebody can comment about how to fix it so that double quotes in the input won't blow things up. For now I worked around it temporarily just to get up and running by adding raw=\"$MSGHDR$MSG\" to the end of my template just to see if it would work OK and it does work OK when you don't have double quotes in there. Matthew.

2 1

Latest and greatest PatternDBs
by Matthew Hall 02 Sep '10

02 Sep '10

Hello all, Thanks to the great help you guys provided earlier I have gotten the hang of patterndb. I am still working on whether I can open-source mine or not, I hope my employer will allow it. I am wondering if this file is the latest and greatest? It's sort of old. http://www.balabit.com/downloads/files/patterndb-snapshot/patterndb-2009120… Thanks, Matthew.

2 1

configure installation path in a on-interactive installation
by Hoenig, Stefan, VF-Group 02 Sep '10

02 Sep '10

Hi there, is it possible to install syslog-ng PE to another path than the default /opt/syslog-ng and use this argument in a non-interactive installation. I tried to configure it by using the .run installer but no option is displayed to change the path. Any ideas or commentrs ??? With best regards / Mit freundlichen Grüßen Stefan Hoenig

2 1

syslog-ng 3.1.2-1 problem/bug/?
by VDR User 02 Sep '10

02 Sep '10

Hi. I'm using Debian testing and "upgraded" from 3.1.1-3 to 3.1.2-1 earlier. Now I'm getting "Error opening file for writing; filename='/logs/vdr.log', error='Permission denied (13)'".. This problem does _not_ occur with 3.1.1-3, which I have downgraded back to (until the problem/bug is solved). Are there any known issues causing this? I don't believe there are any permission problems considering the user owns that dir and it works fine in 3.1.1-3. Following is the diff between the syslog-ng.conf's. The - lines are from my working 3.1.1-3 conf, the + lines are from the 3.1.2-1 conf.... Any help is appreciated. Best regards, Derek --- /etc/syslog-ng/syslog-ng.conf 2010-08-15 12:01:35.000000000 -0700 +++ /etc/syslog-ng/syslog-ng.conf.dpkg-new 2010-08-04 12:07:22.000000000 -0700 @@ -6,7 +6,8 @@ # First, set some global options. options { long_hostnames(off); flush_lines(0); use_dns(no); use_fqdn(no); - owner("root"); group("adm"); perm(0640); + owner("root"); group("adm"); perm(0640); stats_freq(0); + bad_hostname("^gconfd$"); }; ######################## @@ -32,10 +33,10 @@ destination d_auth { file("/var/log/auth.log"); }; destination d_cron { file("/var/log/cron.log"); }; destination d_daemon { file("/var/log/daemon.log"); }; -destination d_kern { file("/var/log/kern.log" template("$FULLDATE $MESSAGE\n")); }; +destination d_kern { file("/var/log/kern.log"); }; destination d_lpr { file("/var/log/lpr.log"); }; destination d_mail { file("/var/log/mail.log"); }; -destination d_syslog { file("/var/log/syslog" template("$FULLDATE $MESSAGE\n")); }; +destination d_syslog { file("/var/log/syslog"); }; destination d_user { file("/var/log/user.log"); }; destination d_uucp { file("/var/log/uucp.log"); }; @@ -106,7 +107,7 @@ local6, local7) and not filter(f_debug); }; filter f_mail { facility(mail) and not filter(f_debug); }; filter f_news { facility(news) and not filter(f_debug); }; -filter f_syslog3 { facility(syslog) and not filter(f_debug); }; +filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug); }; filter f_user { facility(user) and not filter(f_debug); }; filter f_uucp { facility(uucp) and not filter(f_debug); }; @@ -152,14 +153,3 @@ # All messages send to a remote site # #log { source(s_src); destination(d_net); }; - - -######################## -# VDR LOGGING -######################## -destination d_vdr { file("/logs/vdr.log" template("$FULLDATE $MESSAGE\n") owner("test") group("staff") perm(0640)); }; - -filter f_vdr { program("vdr"); }; -filter f_novdr { not program("vdr"); }; - -log { source(s_src); filter(f_vdr); destination(d_vdr); };

2 3

changing incoming facility.level -> kern.emerg
by Charlie Reddington 02 Sep '10

02 Sep '10

I'm having some funky behavior where my central syslog-ng server is marking all forwarded events as facility kern, level emerg. The remote hosts, are CentOS 5 x86_64. They are using the system install syslog. The ones I started with, have nginx web server, patched to send it's log messages under local5.crit for errors, and local5.notice for access logs. It's syslog.conf looks like so. Code: # cat /etc/syslog.conf # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg * # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # nginx local logs local5.notice /usr/ local/logs/access.log local5.crit /usr/ local/logs/error.log # Syslog-ng *.* @remoteIP They are logging locally as expected, and things are coming into the remote host as expected. (from a strace) 32443 recvfrom(3, "<173>nginx: ScrubbedIP www.fa"..., 8192, 0, {sa_family=AF_INET, sin_port=htons(514), sin_addr=inet_addr("ScrubbedIP")}, [16]) = 412 What I can't tell is how they are actually coming in (meaning what level / facility). My central logging server is running the latest syslog-ng. My configs look like... Code: # cat /etc/syslog-ng/syslog-ng.conf # Syslog-ng config options { use_dns(no); keep_hostname(yes); long_hostnames(off); chain_hostnames(off); sync(1); log_fifo_size(1024); create_dirs(yes); perm(0644); stats_freq(3600); mark_freq(600); }; # # Data Sources # # External Syslog source s_external { # udp(); udp(ip("0.0.0.0") port(514) flags(no_parse)); }; # # Where to put Data # # Access Logs destination d_access_log { file("/syslog-ng/$HOST/access.log"); }; # Error Logs destination d_error_log { file("/syslog-ng/$HOST/error.log"); }; # Everything else destination d_log { file("/syslog-ng/$HOST/$FACILITY.$LEVEL"); }; # # Data Filters # # Facility local5 filter f_local5 { facility(local5); }; # Level info filter f_info { level(info); }; # Level notice filter f_notice { level(notice); }; # Level warn filter f_warn { level(warn); }; # Level crit filter f_crit { level(crit); }; # Level err filter f_err { level(err); }; # # Data Logging Locations # # Access Log log { source(s_external); filter(f_local5); filter(f_notice); destination(d_access_log); }; # Error Log log { source(s_external); filter(f_local5); filter(f_crit); destination(d_error_log); }; log { source(s_external); destination(d_log); }; The only problem here is that I'm not getting any access.log or error.log. I'm only getting kern.emerg log with EVERYTHING in it (the last log rule). I have verified that events are coming in correctly as I expect. Code: tcpdump port 514 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 15:47:33.128579 IP scrubbed.syslog > scrubbed.syslog: SYSLOG local5.notice, length: 409 15:47:33.145029 IP scrubbed.syslog > scrubbed.syslog: SYSLOG local5.notice, length: 233 So I'm not sure what syslog-ng is doing to convert this to kern.emerg instead of leaving it local5.notice. At this point, I'm pretty sure the config is sane, but maybe I'm missing something. Syslog-NG / central server stats # /sbin/syslog-ng -V syslog-ng 2.1.4 # cat /etc/redhat-release CentOS release 5.5 (Final) Fetched with GIT. syslog / remote server stats # /sbin/syslogd -v syslogd 1.4.1 # cat /etc/redhat-release CentOS release 5.5 (Final) Any help is appreciated. Charlie

2 1

Re: [syslog-ng] TCP recv bug in syslog-ng v2.09?
by Matthew Hall 19 Aug '10

19 Aug '10

Yeah, now the only problem is all the ancient boxes with no new 12 series code or outdated SMARTNet agreements. But those are a problem regardless of this issue! On Thu, Aug 19, 2010 at 04:05:11PM -0400, Clayton Dukes wrote: > Aha! > Thanks for pointing me to my own company's documentation! hehe. > So, it does appear as though they have provided a fix in later versions of > the code. > That is probably about the best we can ask for, and in my case, it's not a > terrible option since we're running this on IP SLA shadow routers which can > easily be upgraded. > > A HUGE thank you! > > ______________________________________________________________ > > Clayton Dukes > ______________________________________________________________ > > > On Thu, Aug 19, 2010 at 3:50 PM, Matthew Hall <mhall(a)mhcomputing.net> wrote: > > > It's a pretty famous problem among the logging daemon maintainers. > > > > Many logging daemons are struggling with this issue, and they have come > > up with various hacks to try to split out the broken messages using > > timing since they can't always get a '\n'. > > > > Fortunately there is a fix available in recent firmware it appears. > > > > To append a line feed character at the end of each syslog message over > > TCP as delimiter, use the logging delimiter tcp command. To turn off the > > delimiter for Syslog over TCP, use the no form of this command. > > > > logging delimiter tcp > > no logging delimiter tcp > > > > By default, the logging delimiter tcp function is enabled. > > > > This command was introduced in 12.4(24)T. > > > > Usage Guidelines > > > > You can use the logging delimiter tcp to append a line feed character at > > the end of each syslog message over TCP as the delimiter. > > > > Since TCP is a stream protocol, the delimiter helps the external TCP > > syslog listeners to accurately detect the end of each syslog message > > from the incoming TCP stream. In case the external TCP Syslog listeners > > do not work well with the delimiter, use the no form of this command to > > turn it off. > > > > On Thu, Aug 19, 2010 at 03:44:12PM -0400, Clayton Dukes wrote: > > > lol, thanks :-) > > > I'm actually a bit surprised that nobody has said anything before (like > > our > > > customers). > > > Maybe they have, but I did a bug search and didn't find anything. I'll > > have > > > to dig deeper. > > > > > > I know the concept of tcp syslog is still relatively new, so maybe it > > just > > > hasn't come up yet. > > > I'll still have to see if what they have was done on purpose/by design or > > if > > > it is, in fact, a bug (or design flaw?). I dunno, hopefully someone > > smarter > > > than me can answer that, which shouldn't be too hard, heh. > > > > > > > > > ______________________________________________________________ > > > > > > Clayton Dukes > > > ______________________________________________________________ > > > > > > > > > On Thu, Aug 19, 2010 at 3:26 PM, Matthew Hall <mhall(a)mhcomputing.net> > > wrote: > > > > > > > Thanks Clayton, > > > > > > > > I know how to explain things for networking people because I used to > > > > work at HP ProCurve. ;-) > > > > > > > > I really hope you can get this fixed because I think it will cause me > > > > problems at my new job. > > > > > > > > Matthew. > > > > > > > > On Thu, Aug 19, 2010 at 03:13:24PM -0400, Clayton Dukes wrote: > > > > > Matt, > > > > > Thanks for that very good explanation, it's good ammo for me to > > approach > > > > the > > > > > developers @ Cisco with :-) > > > > > > > > > > ______________________________________________________________ > > > > > > > > > > Clayton Dukes > > > > > ______________________________________________________________ > > > > > > > > > > > > > > > On Thu, Aug 19, 2010 at 1:21 PM, Matthew Hall <mhall(a)mhcomputing.net > > > > > > > wrote: > > > > > > > > > > > This understanding is Not quite right. What we are talking about > > here > > > > is > > > > > > the difference between SOCK_STREAM and SOCK_DGRAM socket semantics. > > > > > > > > > > > > In a SOCK_DGRAM each PDU or segment is considered to be a separate > > > > > > atomic message, and each is delivered to the application > > separately. > > > > For > > > > > > a SOCK_DGRAM, a connection just means you have bound the socket to > > a > > > > > > specific remote address. It does not open and close for each > > message > > > > > > because it is not connection oriented. > > > > > > > > > > > > In a SOCK_STREAM each PDU or segment is considered to be additional > > > > > > bytes in a continuous flow of data. The application which receives > > data > > > > > > from a SOCK_STREAM has no way of seeing each PDU or segment > > > > > > individually, because the PDUs or segments are reassembled inside > > the > > > > > > kernel. > > > > > > > > > > > > There is no standard way of simply receiving each SOCK_STREAM > > packet > > > > > > separately because there is no relationship between the size of > > each > > > > > > PDU or segment and the size of the message. The size of the message > > is > > > > > > defined by the '\n'. > > > > > > > > > > > > This difference is also seen in UDP DNS versus TCP DNS. TCP DNS > > sends > > > > > > extra fields in the packets to keep track of the message sizes and > > UDP > > > > > > DNS does not. > > > > > > > > > > > > Matthew. > > > > > > > > > > > > On Thu, Aug 19, 2010 at 11:34:39AM -0400, Clayton Dukes wrote: > > > > > > > bing! (light bulb just went off). > > > > > > > I wasn't thinking about the whole established vs. non established > > > > > > connection > > > > > > > thing. > > > > > > > I couldn't figure out why, if UDP wasn't sending a newline, it > > wasn't > > > > > > > causing the same problem. I wasn't thinking about the fact that > > the > > > > > > > connection was closing, thus ending the stream. > > > > > > > > > > > > > > Thanks for the help! > > > > > > > > > > > > > > ______________________________________________________________ > > > > > > > > > > > > > > Clayton Dukes > > > > > > > ______________________________________________________________ > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 19, 2010 at 11:29 AM, <syslogng(a)feystorm.net> wrote: > > > > > > > > > > > > > > > I explained it already :-) > > > > > > > > When the message comes in over TCP and doesnt end with a > > newline, > > > > > > syslog-ng > > > > > > > > assumes the message is going to be continued in another packet. > > > > When > > > > > > the > > > > > > > > cumulative total of all the messages exceeds the max message > > size > > > > it > > > > > > flushes > > > > > > > > the buffer out and you get all the messages mashed together at > > > > once. > > > > > > > > You can try filing a bug report on bugzilla.balabit.com and > > > > request a > > > > > > new > > > > > > > > flag or something that treats each packet on a tcp source as a > > > > separate > > > > > > > > message, but I'd say the problem is more cisco than syslog-ng > > since > > > > > > > > syslog-ng works fine with all other sources except cisco > > devices > > > > :-/ > > > > > > > > Look at it this way, every thing that sends logs out to tcp > > expects > > > > the > > > > > > > > receiving syslog daemon to treat a packet without a newline as > > a > > > > > > message to > > > > > > > > be continued in a later packet. If syslog-ng changed that > > default > > > > > > behavior, > > > > > > > > all these other things that expect the behavior would break. > > > > > > > > > > > > > > > > -Patrick > > > > > > > > > > > > > > > > > > > > > > > > Sent: Thursday, August 19, 2010 9:12:36 AM > > > > > > > > > > > > > > > > From: Clayton Dukes <cdukes(a)gmail.com> <cdukes(a)gmail.com> > > > > > > > > To: Syslog-ng users' and developers' mailing list > > > > > > > > <syslog-ng(a)lists.balabit.hu> <syslog-ng(a)lists.balabit.hu> > > > > > > > > Subject: Re: [syslog-ng] TCP recv bug in syslog-ng v2.09? > > > > > > > > > > > > > > > > Following up on this: > > > > > > > > > > > > > > > > Interestingly, however, is that when I use tcpdump, I DO see > > each > > > > > > message > > > > > > > > coming in via TCP...so now I'm leaning back towards syslog-ng > > being > > > > the > > > > > > > > problem. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Here's what I can see: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > UDP: > > > > > > > > > > > > > > > > #tcpdump -vvv host 14.3.23.50 -i eth0 > > > > > > > > > > > > > > > > tcpdump: listening on eth0, link-type EN10MB (Ethernet), > > capture > > > > size > > > > > > 96 > > > > > > > > bytes > > > > > > > > > > > > > > > > 09:13:31.465239 IP (tos 0xb8, ttl 251, id 1323, offset 0, flags > > > > [none], > > > > > > > > proto UDP (17), length 134) > > > > > > > > > > > > > > > > 14.3.23.50.51526 > server.x.com.syslog: SYSLOG, length: 106 > > > > > > > > > > > > > > > > Facility local7 (23), Severity notice (5) > > > > > > > > > > > > > > > > Msg: 2975: *Aug 19 12:34:19.465: > > %SYS-5-CONFIG_I[|syslog] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > TCP: > > > > > > > > > > > > > > > > #tcpdump -vvv host 14.3.23.50 -i eth0 > > > > > > > > > > > > > > > > 09:46:29.902063 IP (tos 0x0, ttl 251, id 12253, offset 0, flags > > > > [none], > > > > > > > > proto TCP (6), length 146) > > > > > > > > > > > > > > > > 14.3.23.50.31746 > server.x.com.601: Flags [.], seq > > 233:339, > > > > ack 1, > > > > > > win > > > > > > > > 4128, length 106 > > > > > > > > > > > > > > > > 09:46:29.902077 IP (tos 0x0, ttl 64, id 27779, offset 0, flags > > > > [DF], > > > > > > proto > > > > > > > > TCP (6), length 40) > > > > > > > > > > > > > > > > server.x.com.601 > 14.3.23.50.31746: Flags [.], cksum > > 0xa3bb > > > > > > (correct), > > > > > > > > seq 1, ack 339, win 5840, length 0 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > So, it looks like the syslog message does end, but why is > > syslog-ng > > > > > > > > buffering and showing multiple TCP-based messages as a single > > > > message? > > > > > > > > > > > > > > > > Do I have something misconfigured? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________ > > > > > > > > > > > > > > > > Clayton Dukes > > > > > > > > ______________________________________________________________ > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Aug 17, 2010 at 3:14 PM, Clayton Dukes < > > cdukes(a)gmail.com> > > > > > > wrote: > > > > > > > > > > > > > > > >> yikes! > > > > > > > >> seriously? > > > > > > > >> Guess I'll have to file a bug internally :-) > > > > > > > >> Can someone else positively verify this? > > > > > > > >> Or any suggestions on how I can so that we can recreate it in > > a > > > > lab? > > > > > > > >> > > > > > > > >> ______________________________________________________________ > > > > > > > >> > > > > > > > >> Clayton Dukes > > > > > > > >> ______________________________________________________________ > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> On Tue, Aug 17, 2010 at 2:52 PM, <syslogng(a)feystorm.net> > > wrote: > > > > > > > >> > > > > > > > >>> If I recall correctly its because cisco equipment doesnt > > > > terminate > > > > > > its > > > > > > > >>> log entries with newlines, so when sending via TCP, syslog-ng > > > > thinks > > > > > > the > > > > > > > >>> message is going to be continued in another packet (UDP is > > > > assumed to > > > > > > be 1 > > > > > > > >>> packet per log entry). > > > > > > > >>> The only way to fix this is an ugly hack to set the timeout > > so > > > > that > > > > > > when > > > > > > > >>> it doesnt get a reply within a certain time, it assumes the > > log > > > > entry > > > > > > ended. > > > > > > > >>> but if several log entries are sent within the timeout, then > > > > they'll > > > > > > all be > > > > > > > >>> mashed together into 1 syslog-ng entry. > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> Sent: Tuesday, August 17, 2010 12:28:28 PM > > > > > > > >>> From: Clayton Dukes <cdukes(a)gmail.com> <cdukes(a)gmail.com> > > > > > > > >>> To: Syslog-ng users' and developers' mailing list > > > > > > > >>> <syslog-ng(a)lists.balabit.hu> <syslog-ng(a)lists.balabit.hu> > > > > > > > >>> Subject: [syslog-ng] TCP recv bug in syslog-ng v2.09? > > > > > > > >>> > > > > > > > >>> Hey guys, > > > > > > > >>> Are there any known bugs for syslog-ng v2.09 that won't > > allow a > > > > > > cisco > > > > > > > >>> router to send logs over tcp? > > > > > > > >>> I can see a connection established in syslog-ng. > > > > > > > >>> I also see the message come in via tcpdump, but nothing in > > > > > > syslog-ng's > > > > > > > >>> output. > > > > > > > >>> If I change the router from tcp to udp, messages come in as > > > > expected. > > > > > > > >>> > > > > > > > >>> *Router config:* > > > > > > > >>> > > > > > > > >>> logging source-interface Loopback0 > > > > > > > >>> logging 172.18.224.150 > > > > > > > >>> logging host 172.18.224.190 transport tcp > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> *syslog-ng config:* > > > > > > > >>> > > > > > > > >>> source s_all { > > > > > > > >>> udp(); > > > > > > > >>> tcp(ip(11.31.130.99) port(8002) > > max-connections(300)); > > > > > > > >>> tcp(ip(172.18.224.190) port(601) > > max-connections(300)); > > > > > > > >>> }; > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> *debug output:* > > > > > > > >>> I commented out the line above for the other interface > > > > > > (11.31.130.99), > > > > > > > >>> restarted and this is all I see: > > > > > > > >>> Syslog connection accepted; from='AF_INET(14.3.23.50:63845 > > )', > > > > > > > >>> to='AF_INET(172.18.224.190:601)' > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> *tcpdump:* > > > > > > > >>> > > > > > > > >>> 14:13:46.914566 IP (tos 0x0, ttl 251, id 4303, offset 0, > > flags > > > > > > [none], > > > > > > > >>> proto TCP (6), length 134) > > > > > > > >>> 14.3.23.50.63845 > xxx.com.601: Flags [.], seq 230:324, > > ack > > > > 1, > > > > > > win > > > > > > > >>> 4128, length 94 > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> *Router debug:* > > > > > > > >>> > > > > > > > >>> *Aug 17 17:34:19.772: %SYS-5-CONFIG_I: Configured from > > console > > > > by > > > > > > pnoc > > > > > > > >>> on vty0 (172.18.224.151) > > > > > > > >>> *Aug 17 17:34:20.776: Released port 15205 in Transport Port > > Agent > > > > for > > > > > > > >>> TCP IP type 1 delay 240000 > > > > > > > >>> *Aug 17 17:34:20.776: TCB 0x850F9754 destroyed > > > > > > > >>> *Aug 17 17:34:25.775: TCB83648E60 created > > > > > > > >>> *Aug 17 17:34:25.775: TCB83648E60 setting property TCP_PID > > (8) > > > > > > 845083E4 > > > > > > > >>> *Aug 17 17:34:25.775: TCB83648E60 setting property > > TCP_NO_DELAY > > > > (1) > > > > > > > >>> 845083E8 > > > > > > > >>> *Aug 17 17:34:25.775: TCB83648E60 setting property TCP > > keepalive > > > > > > timeout > > > > > > > >>> (17) 845084A0 > > > > > > > >>> *Aug 17 17:34:25.775: TCP: Random local port generated 63845, > > > > network > > > > > > 1 > > > > > > > >>> *Aug 17 17:34:25.775: TCB83648E60 bound to 14.3.23.50.63845 > > > > > > > >>> *Aug 17 17:34:25.775: Reserved port 63845 in Transport Port > > Agent > > > > for > > > > > > > >>> TCP IP type 1 > > > > > > > >>> *Aug 17 17:34:25.775: TCP: sending SYN, seq 3300233565, ack 0 > > > > > > > >>> *Aug 17 17:34:25.775: TCP0: Connection to 172.18.224.190:601 > > , > > > > > > > >>> advertising MSS 536 > > > > > > > >>> *Aug 17 17:34:25.775: TCP0: state was CLOSED -> SYNSENT > > [63845 -> > > > > > > > >>> 172.18.224.190(601)] > > > > > > > >>> *Aug 17 17:34:25.779: TCP0: state was SYNSENT -> ESTAB [63845 > > -> > > > > > > > >>> 172.18.224.190(601)] > > > > > > > >>> *Aug 17 17:34:25.779: TCP: tcb 83648E60 connection to > > > > > > 172.18.224.190:601, > > > > > > > >>> peer MSS 1460, MSS is 536 > > > > > > > >>> *Aug 17 17:34:25.779: TCB83648E60 connected to > > 172.18.224.190.601 > > > > > > > >>> *Aug 17 17:34:25.779: %SYS-6-LOGGINGHOST_STARTSTOP: Logging > > to > > > > host > > > > > > > >>> 172.18.224.190 port 601 started - reconnection > > > > > > > >>> > > > > > > > >>> > > ______________________________________________________________ > > > > > > > >>> > > > > > > > >>> Clayton Dukes > > > > > > > >>> > > ______________________________________________________________ > > > > > > > >>> > > > > > > > >>> ------------------------------ > > > > > > > >>> > > > > > > > >>> > > > > > > > > > > > > ______________________________________________________________________________ > > > > > > > >>> Member info: > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > > > > >>> Documentation: > > > > > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > > > > > >>> FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > > > > > > ______________________________________________________________________________ > > > > > > > >>> Member info: > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > > > > >>> Documentation: > > > > > > > >>> > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > > > > > >>> FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >> > > > > > > > > ------------------------------ > > > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________________ > > > > > > > > Member info: > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > > > > > Documentation: > > > > > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > > > > > > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________________ > > > > > > > > Member info: > > https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > > > > > Documentation: > > > > > > > > > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > > > > > > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ______________________________________________________________________________ > > > > > > > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > > > > > > > Documentation: > > > > > > http://www.balabit.com/support/documentation/?product=syslog-ng > > > > > > > FAQ: http://www.campin.net/syslog-ng/faq.html > > > > > > > > > > > > > > > > > > > > > > > > >

1 0

How to change timestamp?
by Gregers 18 Aug '10

18 Aug '10

Aug 18 02:16:15 SRC_IP HOST %OrionAlertEngine: "BLA BLA BLA message" I receives this kind of messages quite often. My problem is that it is a perfectly fine header and tag, but the original timestamp is in AM/PM format. How can I force syslog-ng to relay this message with the timestamp from the relaying host(which uses 24 hour format), instead of the sending host? TIA Gregers

2 1

Trying to write syslog to /opt
by Johnny Lapenna 18 Aug '10

18 Aug '10

Hi I'm trying to write the following log to /opt/ in which I am unable to: in syslog-ng.conf I have the following: #filter f_local { facility(local0, local1, local2, local3, # local4, local5, local6, local7); }; filter f_local { facility( local1, local2, local3, local4, local5, local7); }; filter f_sol { facility(local0); }; # Log SOL messages in a separate log destination solmessages { file("/opt/solLOG/sol.log"); }; log { source(src); filter(f_sol); destination(solmessages); }; #destination solmessages { file("/var/log/sol.log"); }; #log { source(src); filter(f_sol); destination(solmessages); }; The above does not work but if I comment out: #destination solmessages { file("/opt/solLOG/sol.log"); }; #log { source(src); filter(f_sol); destination(solmessages); }; destination solmessages { file("/var/log/sol.log"); }; log { source(src); filter(f_sol); destination(solmessages); }; The log is written to /var/log/sol.log, is there a reason why the destination path could only be /var/log/? Thank you in advance. Johnny

2 3

3.1.2 OSE $SEQNUM Documentation Error?
by Mike Civil 17 Aug '10

17 Aug '10

I've been playing with Postgresql and 3.1.2 OSE on Gentoo and found a discrepancy when using the $SEQNUM macro and logging and recording on the same host. The 3.1 (& 3.2) docs state "The sequence number increases for every message, and is not lost even if syslog-ng is reloaded or restarted." However I noticed that each time syslog-ng is restarted the value returned by $SEQNUM starts at 1. A quick check of the source code (specifically templates.c and misc.h) shows no logic to keep state, the value is initialised to 1 on program start. Is this an OSE/PE difference, a documentation hiccup or (more likely!) have I got confused somewhere?

3 5

Bazsi's blog: pdbtool test improvements
by BalaBit Blogs 16 Aug '10

16 Aug '10

2 1

About Syslog-ng error
by Yu Watanabe 16 Aug '10

16 Aug '10

Hello all. I have a question regarding to syslog-ng ver. 2.0.5. I have experienced following error message in the /var/log/message file. Aug 3 04:13:06 host syslog-ng[3772]: Connection broken; time_reopen='5' Aug 3 04:16:45 host syslog-ng[3772]: EOF occurred while idle; fd='39' I found a related article in internet : https://lists.balabit.hu/pipermail/syslog-ng/2007-August/010571.html Following are items that may be relevant to this error: 1. Daily cron rotation using HUP signal every 4:00 am 2. Syslog-ng writes to a physical file to a certain directory 3. Syslog-ng recives log from remote hosts through UDP port Could some one give us an advice of how to cope with this situation? Best Regards, Yu Watanabe

1 0

Syslog-NG OSE : a more and more difficult choice to make.
by Christophe Brocas 16 Aug '10

16 Aug '10

Hello everybody, I really enjoy the syntax, the stability, the flexibility and the so clear and accurate documentation of Syslog-NG OSE. This is why I write this post, I love the product, my message is definitively not a troll. Despite above positive aspects, it is more and more difficult to choose Syslog-NG OSE in corporate environment where you have Linux platforms and others Unix flavors. Rsyslog comes with security and performance features (sql driver, disk based bufferring, Solaris port etc) inside whose can only be acquired through Premium Syslog-NG Edition. If in the future, Rsyslog provides an AIX port on PPC architecture, I really think it will be an ended story for Syslog-NG on corporate environment : it will no more exist a technical reason to stay with an open source under powered solution like Syslog-NG OSE or to buy a solution while it exists an opensource solution with same / more features. I really understand everybody has to earn its life, really. But the current situation in the open source syslog products area is quite difficult for Syslog-NG, that's why I wanted to point the above facts in corporate environment out to you. I don't know how to do : more appliances, more closed products, more consulting ... but the 2 flavors (free and paid) of Syslog-NG are imho an each day harder choice to defend. It is the message from a Syslog-NG user that would like to be able to promote and use it in its company for a long time. Thank you for your reading. Bye Christophe -- Christophe Brocas keyid : 0x237E9DB2 twitter: @cbrocas web : http://brocas.org/blog/ ***************************************************** "Le contenu de ce courriel et ses eventuelles pièces jointes sont confidentiels. Ils s'adressent exclusivement à la personne destinataire. Si cet envoi ne vous est pas destiné, ou si vous l'avez reçu par erreur, et afin de ne pas violer le secret des correspondances, vous ne devez pas le transmettre à d'autres personnes ni le reproduire. Merci de le renvoyer à l'émetteur et de le détruire. Attention : L'Organisme de l'émetteur du message ne pourra être tenu responsable de l'altération du présent courriel. Il appartient au destinataire de vérifier que les messages et pièces jointes reçus ne contiennent pas de virus. Les opinions contenues dans ce courriel et ses éventuelles pièces jointes sont celles de l'émetteur. Elles ne reflètent pas la position de l'Organisme sauf s'il en est disposé autrement dans le présent courriel." ******************************************************

4 4

Pattern extraction
by majid as 15 Aug '10

15 Aug '10

Hi I have problem with pattern extraction from syslog messages. can anyone help me how extract patterns?

5 10

syslog-ng & open core, why we think it is different
by Balazs Scheidler 14 Aug '10

14 Aug '10

Hi, since the email gateway from my new blog doesn't work yet, I copy this post to the list, as again this is an important topic. As you may have seen in my last post, syslog-ng was quoted as an example of "open core" development model, which has problems in the eyes of some Open Source purists. I was trying to do my homework and understand their point of view and see what we can do about it. If you are interested, you can read these articles: * Dave Neary's Rotten to the Open Core * Simon Philip's Open Core is Bad For You * Henrik Ingo's So if I don't call myself 'open source vendor', then everything is fine? (yes) People who know me either personally or through the syslog-ng mailing list would probably know that I by no means want to publish something substandard or cripple the syslog-ng open source functionality in any way. I wanted to start with this statement as the way I see there's only a small and subtle difference between an "open core" and a "true open source" project. I'd like to explain why I feel that syslog-ng is still a "true open source" project. The most to the point explanation why "open core" is bad I saw thus far was: "The question this boild down to is: Does the Open Source version exist because it’s a nice selling point or does it exist because someone believes in the values of Free Software?" [link] I'd say that both is true for BalaBit: we do believe on Free Software and we do publish a lot of that, outside syslog-ng as well (our TProxy work was integrated in kernel 2.6.28, various libdbi/dbi-drivers patches, iptables-utils, jailer, restrict and a couple of others, read this page for more). Some of these are more successful, others less so, but I guess the same applies to other free software hackers too. So why I think there's only a subtle difference between an "open core" and "true open source" development? Because at the fact-side, they look _very_ similar. Some quoted "true open source" projects (Apache, rsyslog, JBoss, Tomcat) and probably others support commercial development on top of them: their "core" license is compatible with commercial licenses (either LGPL or something else). The difference seems to be the very existance of a commercial offering from the same author. So while syslog-ng 3.2 will have the same licensing structure as rsyslog, some still say that syslog-ng is not a true open source project. Again, the root cause for that is the existence of the syslog-ng Premium Edition package. I was trying to understand the logic behind this stance and came up with the following list of reasons (not necessarily applying to syslog-ng, but more on that later): * with "open core" development, products usually fork the Open Source part of the project for their internal development and the flow of bugfixes and features on the open source portion happens in a closed way as a code dump * the development is driven by the commercial edition, changes are trickling to the OSE version with a delay * because of this setup, the community has no say in the features or in the way those features are developed * because of this setup & without extra effort the open core part can quickly deteriorate * the neglection of the open source portion can go as far as being completely unusable (that's why the word crippleware) * this process may not be visible right from the start, Open Core development may look like a true open source at first, but this can happen with time, effectively pushing users to pay money when the OSE version deteriorates and the replacement of a core component of a system would be more expensive * when a competing open source solution is contributed, such a contribution is not accepted into the upstream project, because that would compete with the commercial edition. The way I see it, the line between "open core" / "true open source" is very thin and also difficult to judge for an outsider. These are the reasons why I think that although syslog-ng does have a commercial edition, it is still an open source project. Code base fork Right now the code base for the OSE/PE editions are in different repositories. However one reason of the licensing change in 3.2 is to eliminate this and use the same core in both the PE and the OSE editions. It must also be added, that the current setup was created because of community needs: in the early days, only a single, internal repository was used and the OSE edition was exported as a tarball every night as a snapshot. As the lack of a real version control system was a problem for the community we came to the currently used setup: two git trees, patches synchronized between them. This is of course much more work, but we volunteered to do that. Development drive Right now, the development on syslog-ng happens in three parallel teams. They each create original work. These are: * the syslog-ng Store Box team, * the syslog-ng Premium Edition team and * the Open Source team. The method used to synchronize work is as follows: * bugfixes are done in the code base the bug report is coming from * bugfixes in parallel branches are synced as the first task of every maintenance release This means that it is true, that with PE related features OSE is behind, but this is also symmetric: OSE developed features take their time to propagate into the Premium Edition. One noteworthy thing is that there are several people inside BalaBit who contribute their free time to syslog-ng development, you can find their git trees on git.balabit.hu. And the features in the OSE edition are not worthless, they are really important in the future of syslog-ng. Some examples: * db-parser to parse log messages (in 3.0) * change from syslog-only implementation to a syslog-independent one (nvtable in 3.1) * the new plugin architecture (in 3.2) * syslog-ng configuration library (in 3.2) * first support to non-syslog messages (3.2) * and so on. Community involvement in development decisions Well, this is a though question. syslog-ng may not attract that many contributors or I myself suck at this, but it has nothing to do with the current PE/OSE division. Even if I was directly asking, I received only little feedback and this was in no way different in the first 8 years (1998-2006) of syslog-ng when there was no commercial fork. In fact, the community seems to be more active these days. But anyway, since I myself develop the OSE version and it is out in the open (as the git history and my blog posts prove), the community would have a chance, at least in the features that BalaBit itself has chosen to implement. Also, small feature requests on the mailing list are quite rapidly implemented (the last thing I remember from the last couple of weeks is the "condition()" option to rewrite expressions). With the current architecture/license change, I plan to go back and seek for contributors who didn't want to sign the CLA whether they'd be interested in contributing their code as a plugin, or perhaps directly into syslog-ng. Open Core deterioration I myself worked very hard for this not to happen, and if you look at the mailing list archive, a lot of this time actually was spent from my free time. Since the project/release overhead of a fix is smaller in the case of the Open Source development, it quite often happened that the fix was first made available as a git commit in the OSE branch. Then integrated tested and released in the Premium Edition as well with a slight delay. Forcing users to use the Premium Edition I always said and I believe that the Open Source Edition is a full featured syslog implementation. I'd say nothing proves my word better than that it became and was proposed as the default syslog implementation of various distributions. The existance of the Premium branch even created further features in the OSE, which may or may not have happened without that. It is perfectly acceptable for us, that someone will stick to the Open Source edition. We work hard to update syslog-ng packages in distributions to its latest version, even though that competes with our commercial offering. So I wouldn't say we force them, not on the feature and neither on the quality side of things. The Premium Edition offsers the following added value compared to the Open Source Edition: * support for a wide variety of Opearing System and CPU combinations (about 27 right now, but about 10 new combinations are in the pipeline for 4.0) * longer support cycles * support services * maintenance & bugfixes * .... and additional features. Right now, our feeling is that the additional features is a selling point and without it, we'd be earning less money. With less money, we could allocate less time for syslog-ng. And I'm sure that'd affect syslog-ng, both Open Source and Premium Editions negatively. Competing features One more complaint against the "Open Core" business model is that there's a conflict of interest on the maintainer of the Open Source portion whenever the community is contributing a feature that is present in the commercial edition. The way I see it and with the advent of the new plugin based architecture, this is not necessarily a problem, since a plugin doesn't have to be distributed with the main distribution tarball. In fact I expect several plugins that wouldn't be added to the distribution. For example gyp's Twitter destination is a nice hack, but I probably wouldn't include it. Also, a Linux distribution is in the position that a plugin/functionality can easily reach much more users than the syslog-ng packages that we ourselves publish. But anyway, once a feature is in the scope of syslog-ng (and being a feature in the commercial edition certainly proves that point) and a working implementation is contributed on the syslog-ng mailing list, then I hereby state that we are willing to include it in the main syslog-ng distribution. Of course technical and quality issues do still apply, but if there's enough support from the community (e.g. on the syslog-ng mailing list) and if at least the plugin names are different from the commercial edition (this is purely a technical reason to make the maintenance feasible and user confusion low). Since I feel that this topic is one of the strongest arguments against the "freeness" of syslog-ng, but with the above statements we are at the same level as any other projects "true open source project": you need to take our word for it. I hope that with this post and with our last 12 years of open source work, we earned this trust. And if we don't keep this promise we can be blamed. But until that happens, please don't judge syslog-ng's freeness. Conclusion I agree that trying to balance between the Open Source and Premium Editions of syslog-ng is a difficult task, one that requires a lot of work too. We may have made some mistakes, but our intention is clear: OSE is an independent entity, standing on its own feet. I hope this blog post clears up some of the confusion around this. PS: Thanks for reading until this far. -- Bazsi

2 1

Rewrite engine and backreferences
by mike jack 14 Aug '10

14 Aug '10

Hi list A system XYZ produces "csv-like" logs. Fields are separated by whitespace and different columns have different meanings. The column titles are not included in log messages. I would like to rewrite the log messages so that the column title is prefixed to each column contents. For example: qwerty asdfg zxcvb 12345 would be rewritten to system=qwerty action=asdfg flags=zxcvb result=12345 In perl that would be accomplished by backreferences, e.g. s/(\S+)\s+(\S+)\s+(\S+)\s+(\S+)/system=$1 action=$2 flags=$3 result=$4/ This does not seem to work with syslog-ng rewrite rules out of the box. This is what I have tried: rewrite test_columns{subst("(\S+)\s+(\S+)\s+(\S+)\s+(\S+)", "system=$1 action=$2 flags=$3 result=$4", value("MESSAGE"));}; The rewriting works without regexp, so I assume that the configuration syntax is correct. Is this possible with syslog-ng? If not using rewrite engine, does syslog-ng support feeding the logs through an external script? The system that produces the logs cannot be changed. The syslog-ng version is 3.0.4 Thanks .................................................................... Luukku Plus -paketilla pääset eroon tila- ja turvallisuusongelmista. Hanki Luukku Plus ja helpotat elämääsi. http://www.mtv3.fi/luukku

4 3

Using source tags in rewrites?
by Steve Barnes 14 Aug '10

14 Aug '10

Hi Is it possible to use source tags in rewrite rules? The documentation indicates only macros are acceptable? Using Syslog 3.1.2. Cheers Steve

4 7

[Bug 87] New: csv-parser: specifying drop-invalid and greedy flags always fails.
by bugzilla＠bugzilla.balabit.com 14 Aug '10

14 Aug '10

https://bugzilla.balabit.com/show_bug.cgi?id=87 Summary: csv-parser: specifying drop-invalid and greedy flags always fails. Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: alex-pub.balabit(a)reflexion.net Type of the Report: bug Estimated Hours: 0.0 In this example, I am trying to parse the postfix log statement. Postfix specifies $PROGRAM as a pair separated by slash, e.g "postfix/master" or "postfix/postfix-script". The "postfix" part is configurable, and I would like to parse it out from the $PROGRAM macro. I tried it first with this parser parser p_rfx_postfix_program_parser { csv-parser( columns( "POSTFIX.SERVICE_NAME", "POSTFIX.PROGRAM" ) delimiters( "/" ) flags( drop-invalid, escape-none, greedy ) template( "${PROGRAM}" ) ); }; The intention here is that ${PROGRAM} should always contain at least 1 forward slash, and I only want to parse the part before the first slash. However this parser always fails. Removing greedy flag makes this parser succeed, however it does change the meaning of the action that I am trying to perform. Looking at the logparser.c: log_csv_parser_process, I've noticed that when LOG_CSV_PARSER_GREEDY flag is on, then `cur_column` variable is reset to NULL. Then right before the function ends it checks if `cur_column` is null and LOG_CSV_PARSER_DROP_INVALID flag is on, then fails the function. I think the check at the end of the function should take into account whether GREEDY operation has been performed or not. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

[Bazsi's blog] blog moved
by Bazsi 13 Aug '10

13 Aug '10

I'm moving this blog to a wordpress instance deployed on our company webserver. The URL should be unchanged but the RSS/Atom feeds have changed, so please update your bookmarks. Thanks and sorry for the confusion. -- Posted By Bazsi to Bazsi's blog at 8/12/2010 03:23:00 PM

1 0

Using source tags in rewrites?
by Steve Barnes 10 Aug '10

10 Aug '10

Hi Is it possible to use source tags in rewrite rules? The documentation indicates only macros are acceptable? Cheers Steve

1 0