- syslog-ng - lists.balabit.hu

Fix for bug 162 in an official release yet?
by Mr. Jan Walter 26 Mar '12

26 Mar '12

We're running into the frame header issue a lot in our deployment environment, so I am wondering if the fix, committed the day before the 3.3.4 release, made it into 3.3.4. It's not in the changelog posted, but given the dates, it was worth the question. https://bugzilla.balabit.com/show_bug.cgi?id=162 Did it? Cheers, Jan

2 3

Installing syslog-ng 3.3.4 OSS on multiple Centos servers
by Srinivasan Subramanian 26 Mar '12

26 Mar '12

Hi I have a need to install syslog-ng 3.3.4 on multiple Centos servers. Unfortunately i am unable to find RPMs for the newer versions and only find the source downloads. 1. Any links to contributed repositories for Centos 5.x?2. Alternatively can anyone point me to Source RPMs for Centos 5.x so that I can at least make the RPMs myself? Even EPEL 6 I find only 3.2.x source RPMs RegardsSrini

6 5

RFC: Removing the 'dummy' module
by Gergely Nagy 23 Mar '12

23 Mar '12

Hi! syslog-ng comes with a 'dummy' module, that does nothing, but emit a notice whenever it receives a message. I doubt anyone uses it, because it is useless, and serves as an aid for module writers only, as a skeleton to build on. However, it is built AND installed by default, which is kinda awkward, in my opinion, and perhaps a bit confusing too: when I see a dummy module automatically installed, I start to wonder just what the heck it is good for and why it is there in the first place? Since it's completely useless, I'd like to propose either of the following: 1) Remove the module completely from the source code. The rationale behind this is that it's a very basic example, shows very little about how a destination driver should work, and is even a little bit stale here and there. For example, the grammar directly pokes the driver structure, which is a bad habit (accessors are a whole lot better!). Furthermore, when I wanted to write a driver, I looked at the dummy driver first, but after a couple of minutes, I concluded it has so very little information, that it's not enough for me as a starting point. So I started to look at other drivers, and learned from there. In that sense, it doesn't even succeed at being a reasonable skeleton, or starting point. A HOWTO document, or something similar about the architecture of the various drivers would be much much more useful, but this document is yet to be written. 2) Keep the module, but don't install it, ever. As a compromise, the module could be kept, but not installed. This way we still have a very small example, but it's not installed needlessly. I think it's obvious I'd rather go with the first option, but perhaps there's good reasons for keeping it, that I overlooked. The decision is in Bazsi's hand, anyway. -- |8]

2 1

[Bug 154] New: Processes forked via program() do not properly terminate
by bugzilla＠bugzilla.balabit.com 23 Mar '12

23 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=154 Summary: Processes forked via program() do not properly terminate Product: syslog-ng Version: 3.2.x Platform: Other OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: hendrik.volker(a)de.verizonbusiness.com Type of the Report: --- Estimated Hours: 0.0 I am using a SyslogNG 3.2.4 build for our company internal version of Solaris 10 on some SUN V245 SPARC servers. In a project I am using the following config: ---8<--- source s_whsvoipgwv2_ngrep_bge1 { program ( "/prod/tools/ngrep -d bge1 -t -q -W single SIP port 5060" flags (no-parse) tags ("IF1") ); }; parser p_whsvoipgwv2_ngrep { csv-parser ( columns ("NGREP.PROTO" "NGREP.DATE", "NGREP.TIME", "NGREP.SRCADDR", "NGREP.DUMMY", "NGREP.DSTADDR", "NGREP.MSG" ) flags (drop-invalid, escape-none, greedy, strip-whitespace) delimiters (" ") ); }; template t_whsvoipgwv2_ngrep { template ( "$MONTH_ABBREV $DAY ${NGREP.TIME} $HOST ${NGREP.PROTO} ${NGREP.SRCADDR} ${NGREP.DSTADDR} ${ NGREP.MSG}\n" ); }; destination d_whsvoipgwv2_sip_voipslp { file ("/var/home/voipslp/sip/sip.$YEAR$MONTH$DAY$HOUR.log" owner("voipslp") group("voipsl") perm(0644) template(t_whsvoipgwv2_ngrep) ); }; log { source(s_whsvoipgwv2_ngrep_bge1); parser(p_whsvoipgwv2_ngrep); destination(d_whsvoipgwv2_sip_voipslp); flags (final); }; ---8<--- With the log file rotation every hour it seems that SyslogNG is starting new instances of ngrep, while not terminating the old instances. This results in lots! of running but unused ngreps. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

2 11

[Bug 150] New: Possible memory leak for TLS connections
by bugzilla＠bugzilla.balabit.com 23 Mar '12

23 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=150 Summary: Possible memory leak for TLS connections Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: eric(a)sevenscale.com Type of the Report: --- Estimated Hours: 0.0 I've noticed that memory usage seems to grow as clients using TLS connect and disconnect. I ran syslog-ng under valgrind while I caused a number of clients to connect and disconnect a few times and saved the leak summary from valgrind: https://gist.github.com/893a9687bf98dc53f0e7 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

json configure problem in 3.4
by Peter Czanik 23 Mar '12

23 Mar '12

Hello, First I ran into this on FreeBSD, but now also checked on Linux. If I run configure without arguments, then json is found by configure: JSON support (module) : parser=yes, formatter=yes (using json-c) On the other hand, if I run: ./configure --with-json=json-c Then it fails with: configure: error: Cannot find json-c version >= 0.9: is pkg-config in path? Also, there is no way to disable json support, which is a problem for packagers or the FreeBSD ports system. A "--with-json=none" could solve this. Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

segfault in 3.4 alpha1
by Peter Czanik 23 Mar '12

23 Mar '12

Hello, I tried to compile and run alpha1 on openSUSE 12.1 and FreeBSD 8.1. I got a segfault on start on both. Here are the backtraces: fb81# gdb74 -c syslog-ng.core /usr/local/sbin/syslog-ng GNU gdb (GDB) 7.4 [GDB v7.4 for FreeBSD] Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd8.1". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/sbin/syslog-ng...(no debugging symbols found)...done. [New process 100055] [New Thread 8018041c0 (LWP 100055)] Core was generated by `syslog-ng'. Program terminated with signal 11, Segmentation fault. #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 60 file-perms.c: No such file or directory. (gdb) backtrace #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 #1 0x0000000801c0cda0 in afsocket_parse (lexer=0x80196b000, instance=0x7fffffffc788, arg=0x0) at afsocket-grammar.y:1186 #2 0x000000080068b4f3 in cfg_parser_parse (arg=<optimized out>, instance=<optimized out>, lexer=0x80196b000, self=0x801d16fc0) at cfg-parser.h:83 #3 plugin_parse_config (self=0x801d174b0, cfg=<optimized out>, yylloc=<optimized out>, arg=0x0) at plugin.c:211 #4 0x0000000800698730 in main_parse (lexer=0x80196b000, dummy=0x7fffffffec48, arg=0x0) at cfg-grammar.y:518 #5 0x0000000800669ef9 in cfg_parser_parse (arg=<optimized out>, instance=<optimized out>, lexer=<optimized out>, self=<optimized out>) at cfg-parser.h:83 #6 cfg_run_parser (self=0x801949180, lexer=0x80196b000, parser=0x8007c4140, result=0x7fffffffec48, arg=0x801c0f760) at cfg.c:316 #7 0x000000080066a006 in cfg_read_config (self=0x801949180, fname=0x8006ab16b "/usr/local/etc/syslog-ng.conf", syntax_only=<optimized out>, preprocess_into=0x0) at cfg.c:347 #8 0x0000000800685baa in main_loop_init () at mainloop.c:674 #9 0x000000000040171f in main () (gdb) And the same core file with gdb from the base system: fb81# gdb -c syslog-ng.core /usr/local/sbin/syslog-ng GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... Core was generated by `syslog-ng'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/local/lib/libsyslog-ng-3.4.0alpha1.so...done. Loaded symbols for /usr/local/lib/libsyslog-ng-3.4.0alpha1.so Reading symbols from /usr/local/lib/libnet115/libnet.so.7...done. Loaded symbols for /usr/local/lib/libnet115/libnet.so.7 Reading symbols from /usr/lib/librt.so.1...done. Loaded symbols for /usr/lib/librt.so.1 Reading symbols from /usr/local/lib/libgmodule-2.0.so.0...done. Loaded symbols for /usr/local/lib/libgmodule-2.0.so.0 Reading symbols from /usr/local/lib/libgthread-2.0.so.0...done. Loaded symbols for /usr/local/lib/libgthread-2.0.so.0 Reading symbols from /usr/local/lib/libglib-2.0.so.0...done. Loaded symbols for /usr/local/lib/libglib-2.0.so.0 Reading symbols from /usr/local/lib/libintl.so.9...done. Loaded symbols for /usr/local/lib/libintl.so.9 Reading symbols from /usr/local/lib/libiconv.so.3...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /usr/local/lib/libevtlog.so.0...done. Loaded symbols for /usr/local/lib/libevtlog.so.0 Reading symbols from /usr/local/lib/libpcre.so.1...done. Loaded symbols for /usr/local/lib/libpcre.so.1 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/local/lib/syslog-ng/libconfgen.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libconfgen.so Reading symbols from /usr/lib/libz.so...done. Loaded symbols for /usr/lib/libz.so Reading symbols from /usr/local/lib/syslog-ng/libafsocket.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libafsocket.so Reading symbols from /usr/local/lib/syslog-ng/libsyslog-ng-crypto.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libsyslog-ng-crypto.so Reading symbols from /usr/local/lib/libcrypto.so.7...done. Loaded symbols for /usr/local/lib/libcrypto.so.7 Reading symbols from /usr/local/lib/libssl.so.7...done. Loaded symbols for /usr/local/lib/libssl.so.7 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 60 file-perms.c: No such file or directory. in file-perms.c [New Thread 8018041c0 (LWP 100055)] (gdb) backtrace #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 #1 0x0000000801c0cda0 in afsocket_parse (lexer=0x80196b000, instance=0x7fffffffc788, arg=0x0) at afsocket-grammar.y:1186 #2 0x000000080068b4f3 in plugin_parse_config (self=0x801d174b0, cfg=Variable "cfg" is not available. ) at cfg-parser.h:83 #3 0x0000000800698730 in main_parse (lexer=0x80196b000, dummy=0x7fffffffec48, arg=0x0) at cfg-grammar.y:518 #4 0x0000000800669ef9 in cfg_run_parser (self=0x801949180, lexer=0x80196b000, parser=0x8007c4140, result=0x7fffffffec48, arg=Variable "arg" is not available. ) at cfg-parser.h:83 #5 0x000000080066a006 in cfg_read_config (self=0x801949180, fname=0x8006ab16b "/usr/local/etc/syslog-ng.conf", syntax_only=Variable "syntax_only" is not available. ) at cfg.c:347 #6 0x0000000800685baa in main_loop_init () at mainloop.c:674 #7 0x000000000040171f in main () (gdb) This binary was compiled by copying sysutils/syslog-ng port to syslog-ng-devel, adding --enable-debug to configure args. I ran into one more interesting thing: with json-c enabled in "make config", configure failed with: configure: error: Cannot find json-c version >= 0.9: is pkg-config in path? When disabled json-c in "make config" it was found by configure and was built. The relevant part of "Makefile" is: .if defined(WITH_JSON_GLIB) LIB_DEPENDS+= json-glib-1.0.0:${PORTSDIR}/devel/json-glib CONFIGURE_ARGS+= --enable-json --with-json=json-glib PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .elif defined(WITH_JSON_C) LIB_DEPENDS+= json.0:${PORTSDIR}/devel/json-c CONFIGURE_ARGS+= --enable-json --with-json=json-c PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .else CONFIGURE_ARGS+= --disable-json .endif So, in theory json support was disabled... And another backtrace, this time from openSUSE: (gdb) backtrace #0 0xb7684e8d in cfg_tree_compile_node () from /lib/libsyslog-ng-3.4.0alpha1.so #1 0xb7684f60 in cfg_tree_compile_node () from /lib/libsyslog-ng-3.4.0alpha1.so #2 0xb7685557 in cfg_tree_compile_rule () from /lib/libsyslog-ng-3.4.0alpha1.so #3 0xb7685954 in cfg_tree_compile () from /lib/libsyslog-ng-3.4.0alpha1.so #4 0xb76859ee in cfg_tree_start () from /lib/libsyslog-ng-3.4.0alpha1.so #5 0xb76808c0 in cfg_init () from /lib/libsyslog-ng-3.4.0alpha1.so #6 0xb76a07ea in main_loop_init () from /lib/libsyslog-ng-3.4.0alpha1.so #7 0x080491a7 in main () (gdb) Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 11

afsmtp packaging problems
by Peter Czanik 23 Mar '12

23 Mar '12

Hello, There are a couple of problem with smtp support. Tested on openSUSE and FreeBSD. Neither have libesmtp installed, at least not as default. So configure is right: SMTP support (module) : no On the other hand: - libafsmtp is built and installed - unlike the others, it is installed as a versioned library: linux-v3g3:~ # ls -l /lib/syslog-ng/ total 556 -rwxr-xr-x 1 root root 73212 Mar 21 09:14 libaffile.so -rwxr-xr-x 1 root root 48567 Mar 21 09:14 libafprog.so lrwxrwxrwx 1 root root 18 Mar 21 09:18 libafsmtp.so -> libafsmtp.so.0.0.0 lrwxrwxrwx 1 root root 18 Mar 21 09:18 libafsmtp.so.0 -> libafsmtp.so.0.0.0 -rwxr-xr-x 1 root root 6445 Mar 21 09:14 libafsmtp.so.0.0.0 -rwxr-xr-x 1 root root 118106 Mar 21 09:14 libafsocket.so -rwxr-xr-x 1 root root 34040 Mar 21 09:14 libafuser.so -rwxr-xr-x 1 root root 23092 Mar 21 09:14 libbasicfuncs.so -rwxr-xr-x 1 root root 11819 Mar 21 09:14 libconfgen.so -rwxr-xr-x 1 root root 38464 Mar 21 09:14 libcsvparser.so -rwxr-xr-x 1 root root 105045 Mar 21 09:14 libdbparser.so -rwxr-xr-x 1 root root 33668 Mar 21 09:14 libdummy.so -rwxr-xr-x 1 root root 21134 Mar 21 09:14 libsyslogformat.so -rwxr-xr-x 1 root root 28448 Mar 21 09:14 libsyslog-ng-crypto.so -rwxr-xr-x 1 root root 7339 Mar 21 09:14 libtfuuid.so ToDo: - option to disable smtp support - do not build/install it if build dependency is unavailable - do not install it as a versioned library Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

Syslog-NG forwarding the messages to a Remote TCP port with more than one message in a single packet
by anji prassana 18 Mar '12

18 Mar '12

I am using syslog-ng *version * [root@Cypher-210 ~]# /usr/local/sbin/syslog-ng -V syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Mar 6 2012 13:06:17 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,parser,syslogformat Available-Modules: afprog,convertfuncs,affile,afsocket-tls,confgen,csvparser,syslogformat,dbparser,basicfuncs,afmongodb,dummy,afuser,afsocket Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on Problem: ------------ Syslog-ng is forwarding the TCP packets to a Remote TCP port with More than one message in a single TCP Packet.But, in between the messages inside a packet, It doesn't includes any delimiter like "\n" or some other.But, the receiving Program listens on TCP requires a newline as a delimiter b/w messages so as to parse the messages individually. But, As Syslog-ng doesn't includes a separator b/w messages in a packet, The Receiver simply considering whole messages in a packet as a single Message which leads to false and unacceptable parsing. Please look into the following messages exists in a single TCP packet.The one I colored red is a start of new message. root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shadow *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/head -n 4 /etc/inittab *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/tail -n 1 *<13>*mar 6 17:22:15 cypher-210 sshd[5583]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:15 cypher-210 sshd[5765]: accepted password for root from 10.0.15.218 port 44258 ssh2 <13>mar 6 17:22:15 cypher-210 sshd[5765]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:16 cypher-210 sshd[5278]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 pam_timestamp_check: pam `/' permissions are lax <13>mar 6 17:22:17 cypher-210 sshd[5281]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shad These are the settings i made in the syslog-ng.conf file: source s_test { internal(); unix-stream("/dev/log"); udp(); file("/var/log/syslog-ng_local"); }; destination d_test { tcp("10.0.15.18" port(9500)); #My Program listening on tcp port 9500 requires newline as a separator for each message. }; log { source(s_test); destination(d_test); }; Can anyone kindly help me, 1. How can i limit one tcp packet to hold only one message? [or] 2. How can I add newline as a delimiter b/w each message before the packet(s) are forwarding to a destination so that it could parse properly? Your Help is really appreciate.. Thanks&Regards Anjaneyulu P

3 3

Packet fragmentation issue
by José Moreno 18 Mar '12

18 Mar '12

Hi all, I'm running syslog-ng 2.4.1, log sources send to a log server which beside keeping the original data as is in files, forwards them in real time to a SIEM, spoofing source IP. My problem comes after some logs are too long to fit in a single frame, log server fragments those packets when sending them to SIEM and spoofing is not performed for them. Enviado desde mi iPhone

4 5

[Bug 165] New: macro sequenceId is empty
by bugzilla＠bugzilla.balabit.com 18 Mar '12

18 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=165 Summary: macro sequenceId is empty Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: whille(a)163.com Type of the Report: bug Estimated Hours: 0.0 i'm using syslog-ng both as client and server, the problem is $.SDATA.meta.sequenceId is empty in server. is it client or server reason? any configure modify for this tag? conf: ... #destination d_mydest { file("/var/log/mylog.log" template("${.SDATA.meta.sequenceId}____$MSG\n"));}; server version:3.3.4 client version:3.2.2 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Losing to much remote sent logs
by Daniel Neubacher 18 Mar '12

18 Mar '12

Hello there, I've started playing around with syslog-ng 3.3.4 ose a few days ago but I'm still experiencing some trouble. First of all we want to use syslog-ng to send all of our logs via udp to a central syslog server. This includes of course syslogs, apache logs and custom generated applogs. These logs are generated from 400 clients and produces a minimum of 300 mio. log lines a day. The problem is really simple: I'm losing log lines :P Most of the time everything goes well but when the logs are peaking high 1-5% logs are getting lost. Last night the stats of the server and a client said 0 drops but when I counted the lines I found lost lines. The server has 24g ram & 8 cores and I can rule out a network problem for sure. So now to my questions, has anyone else an idea where I can tweak my cfg or where I have to look to find more clues? Is tcp the only way to get around it? I've attached my syslog server cfg. The so_rcvbuf buffer is the same size as the os net.core.rmem settings. And as described in the various balabit blog posts I played around with log_fetch_limit and flush_lines already. syslog-ng.conf: @version: 3.3 options { threaded(yes); owner("root"); group("root"); perm(0660); dir_owner("root"); dir_group("root"); dir_perm(0770); create_dirs(yes); stats_freq(600); stats_level(2); chain_hostnames(yes); normalize_hostnames(yes); check_hostname(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(60); log_msg_size(16384); log_fifo_size(100000); use_fqdn(yes); #disabled 4 debugging # flush_lines(200); }; source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); }; source s_net { udp( log_fetch_limit(400) so_rcvbuf(51200000) keep_hostname(yes) keep_timestamp(no) ip("10.8.4.10") port(514) ); tcp( so_rcvbuf(51200000) so_keepalive(yes) keep_hostname(no) keep_timestamp(no) ip("10.8.4.10") port(514) ); syslog(); }; filter f_syslog { not program(access.log) and not program(error.log) and not program(beetle.log) and not program(edge.log); }; filter f_apache { program(access.log) or program(error.log); }; filter f_applogs { program(beetle.log) or program(edge.log); }; template t_plain { template("$MSG\n"); template_escape(no); }; destination d_messages { file("/var/log/messages"); }; destination d_remote { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); }; destination d_apache { file("/log/apache/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); }; destination d_applogs { file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); }; log { source(s_src); destination(d_messages); }; log { source(s_net); filter(f_syslog); destination(d_remote); }; log { source(s_net); filter(f_apache); destination(d_apache); }; log { source(s_net); filter(f_applogs); destination(d_applogs); }; Thanks Daniel Neubacher

5 11

using syslog-ng and parsing data from both Windows XP and Windows 7 machines
by Mary A Waddick 18 Mar '12

18 Mar '12

Hi, I am using nxlog to send data from both Windows XP and Windows 7 machines to a Unix machine using syslog-ng . My nxlog.conf files are configured to send im_mseventlog data for the Windows XP boxes and im_msvistalog data for the Windows 7 boxes. (See attached file: new nxlog.conf) Therefore I get slightly different data for each machine. The examples on your website don't show me how to parse out all of the im_mseventlog or im_msvistalog data from the different columns in msg. Can you help me with getting the data? My syslog-ng.conf looks like this. I was told that the eventlog data would have the columns that I included in my table, but I am unable to figure out how to pull that data out of the msg column using the provided macros. I have searched and searched, but have not found any examples. (See attached file: syslog-ng.conf) Thank you for your help. Mary Anne Waddick Raytheon Technical Services Senior Software Engineer II (317) 306-2691 (desk)

2 1

syslog-ng blocking application
by Ulrich David 16 Mar '12

16 Mar '12

Hi, I have an ISC-DHCP who is logging on a local facility on a chrooted system : #options { # chain_hostnames(no); # use_dns(no); # flush_lines(1); # stats_freq(3600); #}; #source dhcp_src { # unix-stream("/chroot/dhcp/dev/log" max-connections(256)); #}; My syslog-ng take logs from this source, filter on program and then put the logs on two destinations : #filter f_chroot_l2 { # program(dhcpd); #}; # #destination dhcp_log { file("/var/log/dhcpd.log" perm(0644)); }; #destination dhcp_sql { program("/usr/local/sbin/dhcp_log_to_mysql"); }; # #log { # source(dhcp_src); # filter(f_chroot_l2); # destination(dhcp_log); # destination(dhcp_sql); # flags(final ); #}; Everything goes well… until I block my program "dhcp_log_to_mysql" (I do it with a lock on the mysql table it uses or with a sleep in the script - for test). When "dhcp_log_to_mysql" is blocked, logs continues to be written on dhcp_log for some minutes/seconds then it stops. My chrooted dhcpd hangs on that too. I don't know why syslog-ng is not just dropping logs (continuing eating them on the source). I don't have flow-control enabled… and I find nothing which can explain the problem. I try to resize the buffer output with very big number (log_fifo_size(45000000)) but it changes nothing. Logs and dhcpd hangs at the same moment. What can I do? Regards David

3 4

delay in forwarding logs
by Ivanov, Konstantin 16 Mar '12

16 Mar '12

I have two servers running syslog-ng 2.0.9 on Dell R710's. I did some benchmarking with loggen [1] and was able to generate 41000 msg/sec locally. When I try to forward log messages from one server to the other I only get about 60 msg/sec. tcpdump shows that only that many are sent over the wire as well. I test this by writing to the log file from a script. Any idea what can be the problem? Here are both configs from the sending and receiving servers: ----------- Sending syslog-ng server -------------- options { log_fifo_size(120000); time_reopen (1); time_reap(1); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); group(adm); perm(0640); dir_perm(0755); keep_hostname (yes); log_fetch_limit(60000); }; source s_zxtm { file("/var/log/zxtm/access_log",follow_freq(1), flags(no-parse)); }; destination d_logrelay { tcp(somehost.com port(514)); }; log { source(s_zxtm); destination(d_logrelay); flags(flow-control); }; --------------- Receiving syslog-ng server ------------------ options { log_fifo_size(3000000); time_reopen (1); time_reap(1); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); group(adm); perm(0640); dir_perm(0755); keep_hostname (yes); stats_freq(600); sync(0); }; source s_zxtm { tcp(max-connections(500), port(514)); }; destination d_zxtm { file("/var/log/zxtm/incoming/$YEAR$MONTH$DAY$HOUR-access_log.aggregated" owner("glassfish") perm(0644) template("$MESSAGE\n") template_escape(no)); }; log { source(s_zxtm); destination(d_zxtm); flags(flow-control); }; -------------- [1]. loggen -iS -r 10000000 -s 200 -I 60 localhost 514 average rate = 41053.50 msg/sec, count=410535

2 1

hostname issue with aggregation from aggregator of end-system
by Ian Veach 16 Mar '12

16 Mar '12

We're currently running loghosts/aggregators for each data center and two OS types. This works fine. We are now trying to forward all syslogs from our aggregators to another single aggregator (a security loghost, sec-loghost, which is proprietary). We are having a problem retaining hostname when forwarding from a loghost aggregator to ANOTHER loghost - but only from one OS type / syslog-ng install. We've been running syslog-ng for some time on RHEL (and previously SLES) and AIX, in a loghost configuration, with no issues. Our current architecture (example): data center 1: rhelhostX -> rhel-loghost1 aixhostX -> aix-loghost1 data center 2: rhelhostX -> rhel-loghost2 aixhostX -> aix-loghost2 aix-loghost's are running syslog-ng 3.0.5, RHEL loghost's are running syslog-ng 3.1.4. Both are similarly configured (see below), and are working fine as advertised by themselves - e.g. each loghost gets $HOST_FROM and properly stores the logs for that host in our architecture (/logs/$YEAR/$MONTH/$DAY/$HOST_FROM/). However, given data center 1, if syslogs follow this path: aixhostX -> aix-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> sec-loghost rhelhostX -> rhel-loghost1 -> rhel-loghost2 sec-loghost sees syslogs from all aixhostXs (even though they passed through aix-loghost1) correctly. However, sec-loghost (and we also tested going to rhel-loghost2) sees ALL syslog entries as coming from rhel-loghost, even though it's simply aggregating from rhelhostX's. loghost1's all correctly get $HOST_FROM and see syslog record hosts correctly. This issue only occurs when forwarding again to another loghost. Our nutshell configuration on BOTH loghosts (related to this) is: options { chain_hostnames(no); keep_hostname(yes); use_fqdn(no); use_dns(yes); }; Any ideas on how to fix this? The only hint I've gotten is from our networks group, who uses kiwi syslog and had to make sure a checkbox for RFC 3164 headers compliance was turned on. I don't see an equivalent of that not already set with the above config. Thanks, Ian

2 1

Profiling syslog-ng memory usage and performance
by William Sweat 16 Mar '12

16 Mar '12

Hello, I'm having issues with syslog-ng hitting swap on a set of servers that handle a lot of web traffic. Are there good techniques to figure out an optimal configuration or if there's something with syslog-ng that may prove problematic with large amounts of requests? I'm using the syslog-ng premium client (LTS v4.0.3b), on Linux systems with 256GB of Memory and RAID10. I've enabled more informational logging, but because each system handles multiple gigabytes of traffic per day, debugging is problematic as this is a problem that manifests after a day (or three). Also I am using nightly logrotate, so syslog-ng does get restarted everyday to write new log files. These are only high traffic servers that are experiencing swap issues, the syslog-ng server doesn't have any problems. Thanks, Will Sweat Demand Media ________________________________ Please NOTE: This electronic message, including any attachments, may include privileged, confidential and/or inside information owned by Demand Media, Inc. Any distribution or use of this communication by anyone other than the intended recipient(s) is strictly prohibited and may be unlawful. If you are not the intended recipient, please notify the sender by replying to this message and then delete it from your system. Thank you.

2 1

Compilation errors syslog-ng 3.2.5 on solaris 10
by Arya, Manish Kumar 15 Mar '12

15 Mar '12

Hi Guys, I have compiled eventlog-0.2.12 and glib-2.12.12 in /opt/syslog/eventlog and /opt/syslog/glib directories. now when I try to compile syslog-ng with following options in configure ./configure --prefix=/opt/syslog/syslog-ng --enable-ipv6 EVTLOG_LIBS=/opt/syslog/eventlog/lib EVTLOG_CFLAGS=/opt/syslog/eventlog/include GLIB_CFLAGS=/opt/syslog/glib/include GLIB_LIBS=/opt/syslog/glib/lib PKG_CONFIG_PATH=/opt/syslog/glib/lib/pkgconfig I see this error checking for GLIB... yes checking sanity checking Glib headers... no configure: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH here is the errors o/p from config.log bash-3.00# grep -i error config.log /usr/include/stdbool.h:42:2: #error "Use of <stdbool.h> is valid only in a c99 compilation environment." conftest.c:59: error: syntax error before "text" conftest.c:64: error: 'for' loop initial declaration used outside C99 mode conftest.c:64: error: `text' undeclared (first use in this function) conftest.c:64: error: (Each undeclared identifier is reported only once conftest.c:64: error: for each function it appears in.) conftest.c:108: error: `false' undeclared (first use in this function) conftest.c:112: error: `true' undeclared (first use in this function) conftest.c:113: error: syntax error before "newvar" conftest.c:123: error: 'for' loop initial declaration used outside C99 mode conftest.c:139: error: `newvar' undeclared (first use in this function) ld: fatal: Symbol referencing errors. No output written to conftest | static void yy_fatal_error YY_PROTO(( yyconst char msg[] )); | if ( c == EOF && ferror( yyin ) ) \ | YY_FATAL_ERROR( "input in flex scanner failed" ); \ | && ferror( yyin ) ) \ | YY_FATAL_ERROR( "input in flex scanner failed" ); | /* Report a fatal error. */ | #ifndef YY_FATAL_ERROR | #define YY_FATAL_ERROR(msg) yy_fatal_error( msg ) | YY_FATAL_ERROR( | "fatal flex scanner internal error--no action found" ); | YY_FATAL_ERROR( | "fatal flex scanner internal error--end of buffer missed" ); | YY_FATAL_ERROR( | YY_FATAL_ERROR( | "fatal error - scanner input buffer overflow" ); | YY_FATAL_ERROR( "flex scanner push-back overflow" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_create_buffer()" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_create_buffer()" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_scan_buffer()" ); | YY_FATAL_ERROR( "out of dynamic memory in yy_scan_bytes()" ); | YY_FATAL_ERROR( "bad buffer in yy_scan_bytes()" ); | YY_FATAL_ERROR( | YY_FATAL_ERROR( "start-condition stack underflow" ); | static void yy_fatal_error( yyconst char msg[] ) | static void yy_fatal_error( msg ) ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. configure:12119: gcc -std=gnu99 -c -g -O2 -Wall -pedantic -Werror -Wno-pointer-sign conftest.c >&5 cc1: error: unrecognized command line option "-Wno-pointer-sign" cc1: error: unrecognized command line option "-Wno-pointer-sign" conftest.c:30: error: size of array `off_t_is_large' is negative conftest.c:41: error: structure has no member named `tm_gmtoff' conftest.c:41: error: structure has no member named `tm_gmtoff' ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. ld: fatal: Symbol referencing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. configure:13336: $PKG_CONFIG --exists --print-errors "dbi >= $LIBDBI_MIN_VERSION" configure:13352: $PKG_CONFIG --exists --print-errors "dbi >= $LIBDBI_MIN_VERSION" ld: fatal: File processing errors. No output written to conftest | /* Override any GCC internal prototype to avoid an error. conftest.c:46: error: `GLIB_SIZEOF_LONG' undeclared (first use in this function) conftest.c:46: error: (Each undeclared identifier is reported only once conftest.c:46: error: for each function it appears in.) configure:13605: error: Glib headers inconsistent with current compiler setting. You might be using 32 bit Glib with a 64 bit compiler, check PKG_CONFIG_PATH bash-3.00# help me !! Regards, -Manish

1 2

[PATCH] value-pairs: Fix a double free on parse errors
by Gergely Nagy 14 Mar '12

14 Mar '12

In the command-line parser code, when we encountered an error, we explicitly free'd the value-pairs structure, but did not return. A few lines later, we called vp_cmdline_parse_rekey_finish(), which also tried to free the same structure. Instead, we should only call _rekey_finish() if we didn't NULL out the value pairs structure earlier. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/value-pairs.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/lib/value-pairs.c b/lib/value-pairs.c index 12cf831..912164f 100644 --- a/lib/value-pairs.c +++ b/lib/value-pairs.c @@ -678,7 +678,8 @@ value_pairs_new_from_cmdline (GlobalConfig *cfg, vp = NULL; } g_option_context_free (ctx); - vp_cmdline_parse_rekey_finish (user_data_args); + if (vp) + vp_cmdline_parse_rekey_finish (user_data_args); return vp; } -- 1.7.9.1

2 2

segfault upon starting
by Jenei Gábor 14 Mar '12

14 Mar '12

Hello, On friday I said about this problem already, but now I could captured some messages by rsyslog(it has other logfiles set manually than syslog-ng). So, syslog-ng sends the following messages upon its init: Mar 4 11:32:18 gabor-netbook supervise/syslog-ng[18834]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='11' Mar 4 11:32:19 gabor-netbook kernel: [ 3039.971733] syslog-ng[21594]: segfault at 8f95000 ip b74ce758 sp bfd86e88 error 4 in libc-2.13.so[b73be000+153000] Do you know what the problem is? Maybe I should disable linux capabilities?(actually I am using linux mint, so it should work though) I'm using version 3.4 complied on my own, so there maybe some configuration issues. Thanks for your help, Gabor

3 2

Excluding the local host
by Clayton Dukes 14 Mar '12

14 Mar '12

Hey all, Seems like it would be simple to exclude the local host, but I can't find anything in the docs. What is the best way to do this? ______________________________________________________________ Clayton Dukes ______________________________________________________________

3 2

patterndb can't match empty log strings
by Kurt Yoder 13 Mar '12

13 Mar '12

All, Sometimes an empty log message is emitted, but I can't match it using pdbtool. I've tried patterns such as: <pattern>@ANYSTRING@</pattern> <pattern></pattern> <pattern> </pattern> Nothing works. Other log messages from the same application are definitely being matched by the @ANYSTRING@, so I know my rule is working. So is there a way to specify an empty log string? Do I have to use a regex instead? Thanks, -Kurt

1 0

unable to use multiple application patterns per ruleset
by Kurt Yoder 13 Mar '12

13 Mar '12

Hi list, Using patterndb, has anyone successfully used multiple application patterns per ruleset? I have the following: <?xml version='1.0' encoding='UTF-8'?> <patterndb version='4' pub_date='2012-03-08'> <ruleset name='rdc-pam' id='rdc-pam'> <patterns> <pattern>blahblah</pattern> <pattern>sshd</pattern> </patterns> <rules> <rule class='system' id='pam_session_opened_for_user_by_uid' provider='rdc'> <description></description> <patterns> <pattern>pam_unix(sshd:session): session opened for user kurt by (uid=0)</pattern> </patterns> <values> </values> <tags> <tag>rdc.alert.ok</tag> </tags> </rule> </rules> </ruleset> </patterndb> As soon as I remove "<pattern>blahblah</pattern>", or move it below "<pattern>sshd</pattern>", this rule starts matching events. When I add it back, it stops matching events. The documentation at http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.2-gui… says (incidentally, the closing "<patterns>" in this example should instead read "</patterns>"): " Specifying multiple patterns is useful if two or more applications have different names (that is, different $PROGRAM fields), but otherwise send identical log messages. <patterns> <pattern>firstapplication</pattern> <pattern>otherapplication</pattern> <patterns> This is exactly what I am trying to do, but it does not appear to work as documented. I manually backported my syslog-ng from Ubuntu Oneiric: $ syslog-ng --version syslog-ng 3.2.4 Installer-Version: 3.2.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#ef7b91e4a1b1f9628c66138b4ae83de7e4c697c6 Compile-Date: Jan 19 2012 02:57:58 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: on Enable-Pacct: off Can anyone offer a suggestion toward making this work? Thanks

1 1

[Bug 160] New: memory leak in message buffering
by bugzilla＠bugzilla.balabit.com 12 Mar '12

12 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=160 Summary: memory leak in message buffering Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 Created an attachment (id=47) --> (https://bugzilla.balabit.com/attachment.cgi?id=47) tunable read rate program destination When writing to a destination that requires syslog-ng to buffer messages, the memory footprint of syslog-ng grows endlessly, even when there is sufficient time to flush the buffers. I am including a read rate tunable perl program destination that I use to slow the destination down. I am including a syslog-ng.conf file I am including the output of valgrind. I run syslog-ng, cat a /var/messages file (100,000 lines) to the input pipe. valgrind now shows definitely lost: 2,620,649 bytes in 4,439 blocks -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

[Bug 162] New: Invalid frame header; header=''
by bugzilla＠bugzilla.balabit.com 12 Mar '12

12 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=162 Summary: Invalid frame header; header='' Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: yuri(a)fused.com Type of the Report: bug Estimated Hours: 0.0 Got always reproducible bug when sending huge logs between machines. Here are the steps: * setup default OSE installs of client & server on rhel5 * configure client to read file("/tmp/input" flags(no-parse)) and send it to syslog(transport("tcp")) * configure server to listen on syslog(transport(tcp)) and store messages in file() * Now let client send big chunk of data: ruby -e 'loop { puts (97+rand(26)).chr*250 }' | head -1000 >> /tmp/input * See on server the following message: syslog-ng[xxx]: Invalid frame header; header='' server's RSTing the connection afterwards It doesn't happen when you start with a couple of lines and then raise the "chunk" to 100-1000 messages per insertion. If you need more info, ask. Thanks. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6