- syslog-ng - lists.balabit.hu

json configure problem in 3.4
by Peter Czanik 23 Mar '12

23 Mar '12

Hello, First I ran into this on FreeBSD, but now also checked on Linux. If I run configure without arguments, then json is found by configure: JSON support (module) : parser=yes, formatter=yes (using json-c) On the other hand, if I run: ./configure --with-json=json-c Then it fails with: configure: error: Cannot find json-c version >= 0.9: is pkg-config in path? Also, there is no way to disable json support, which is a problem for packagers or the FreeBSD ports system. A "--with-json=none" could solve this. Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

segfault in 3.4 alpha1
by Peter Czanik 23 Mar '12

23 Mar '12

Hello, I tried to compile and run alpha1 on openSUSE 12.1 and FreeBSD 8.1. I got a segfault on start on both. Here are the backtraces: fb81# gdb74 -c syslog-ng.core /usr/local/sbin/syslog-ng GNU gdb (GDB) 7.4 [GDB v7.4 for FreeBSD] Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-portbld-freebsd8.1". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/local/sbin/syslog-ng...(no debugging symbols found)...done. [New process 100055] [New Thread 8018041c0 (LWP 100055)] Core was generated by `syslog-ng'. Program terminated with signal 11, Segmentation fault. #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 60 file-perms.c: No such file or directory. (gdb) backtrace #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 #1 0x0000000801c0cda0 in afsocket_parse (lexer=0x80196b000, instance=0x7fffffffc788, arg=0x0) at afsocket-grammar.y:1186 #2 0x000000080068b4f3 in cfg_parser_parse (arg=<optimized out>, instance=<optimized out>, lexer=0x80196b000, self=0x801d16fc0) at cfg-parser.h:83 #3 plugin_parse_config (self=0x801d174b0, cfg=<optimized out>, yylloc=<optimized out>, arg=0x0) at plugin.c:211 #4 0x0000000800698730 in main_parse (lexer=0x80196b000, dummy=0x7fffffffec48, arg=0x0) at cfg-grammar.y:518 #5 0x0000000800669ef9 in cfg_parser_parse (arg=<optimized out>, instance=<optimized out>, lexer=<optimized out>, self=<optimized out>) at cfg-parser.h:83 #6 cfg_run_parser (self=0x801949180, lexer=0x80196b000, parser=0x8007c4140, result=0x7fffffffec48, arg=0x801c0f760) at cfg.c:316 #7 0x000000080066a006 in cfg_read_config (self=0x801949180, fname=0x8006ab16b "/usr/local/etc/syslog-ng.conf", syntax_only=<optimized out>, preprocess_into=0x0) at cfg.c:347 #8 0x0000000800685baa in main_loop_init () at mainloop.c:674 #9 0x000000000040171f in main () (gdb) And the same core file with gdb from the base system: fb81# gdb -c syslog-ng.core /usr/local/sbin/syslog-ng GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols found)... Core was generated by `syslog-ng'. Program terminated with signal 11, Segmentation fault. Reading symbols from /usr/local/lib/libsyslog-ng-3.4.0alpha1.so...done. Loaded symbols for /usr/local/lib/libsyslog-ng-3.4.0alpha1.so Reading symbols from /usr/local/lib/libnet115/libnet.so.7...done. Loaded symbols for /usr/local/lib/libnet115/libnet.so.7 Reading symbols from /usr/lib/librt.so.1...done. Loaded symbols for /usr/lib/librt.so.1 Reading symbols from /usr/local/lib/libgmodule-2.0.so.0...done. Loaded symbols for /usr/local/lib/libgmodule-2.0.so.0 Reading symbols from /usr/local/lib/libgthread-2.0.so.0...done. Loaded symbols for /usr/local/lib/libgthread-2.0.so.0 Reading symbols from /usr/local/lib/libglib-2.0.so.0...done. Loaded symbols for /usr/local/lib/libglib-2.0.so.0 Reading symbols from /usr/local/lib/libintl.so.9...done. Loaded symbols for /usr/local/lib/libintl.so.9 Reading symbols from /usr/local/lib/libiconv.so.3...done. Loaded symbols for /usr/local/lib/libiconv.so.3 Reading symbols from /usr/local/lib/libevtlog.so.0...done. Loaded symbols for /usr/local/lib/libevtlog.so.0 Reading symbols from /usr/local/lib/libpcre.so.1...done. Loaded symbols for /usr/local/lib/libpcre.so.1 Reading symbols from /lib/libthr.so.3...done. Loaded symbols for /lib/libthr.so.3 Reading symbols from /lib/libc.so.7...done. Loaded symbols for /lib/libc.so.7 Reading symbols from /usr/local/lib/syslog-ng/libconfgen.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libconfgen.so Reading symbols from /usr/lib/libz.so...done. Loaded symbols for /usr/lib/libz.so Reading symbols from /usr/local/lib/syslog-ng/libafsocket.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libafsocket.so Reading symbols from /usr/local/lib/syslog-ng/libsyslog-ng-crypto.so...done. Loaded symbols for /usr/local/lib/syslog-ng/libsyslog-ng-crypto.so Reading symbols from /usr/local/lib/libcrypto.so.7...done. Loaded symbols for /usr/local/lib/libcrypto.so.7 Reading symbols from /usr/local/lib/libssl.so.7...done. Loaded symbols for /usr/local/lib/libssl.so.7 Reading symbols from /libexec/ld-elf.so.1...done. Loaded symbols for /libexec/ld-elf.so.1 #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 60 file-perms.c: No such file or directory. in file-perms.c [New Thread 8018041c0 (LWP 100055)] (gdb) backtrace #0 file_perm_options_set_file_perm (self=0x0, file_perm=384) at file-perms.c:60 #1 0x0000000801c0cda0 in afsocket_parse (lexer=0x80196b000, instance=0x7fffffffc788, arg=0x0) at afsocket-grammar.y:1186 #2 0x000000080068b4f3 in plugin_parse_config (self=0x801d174b0, cfg=Variable "cfg" is not available. ) at cfg-parser.h:83 #3 0x0000000800698730 in main_parse (lexer=0x80196b000, dummy=0x7fffffffec48, arg=0x0) at cfg-grammar.y:518 #4 0x0000000800669ef9 in cfg_run_parser (self=0x801949180, lexer=0x80196b000, parser=0x8007c4140, result=0x7fffffffec48, arg=Variable "arg" is not available. ) at cfg-parser.h:83 #5 0x000000080066a006 in cfg_read_config (self=0x801949180, fname=0x8006ab16b "/usr/local/etc/syslog-ng.conf", syntax_only=Variable "syntax_only" is not available. ) at cfg.c:347 #6 0x0000000800685baa in main_loop_init () at mainloop.c:674 #7 0x000000000040171f in main () (gdb) This binary was compiled by copying sysutils/syslog-ng port to syslog-ng-devel, adding --enable-debug to configure args. I ran into one more interesting thing: with json-c enabled in "make config", configure failed with: configure: error: Cannot find json-c version >= 0.9: is pkg-config in path? When disabled json-c in "make config" it was found by configure and was built. The relevant part of "Makefile" is: .if defined(WITH_JSON_GLIB) LIB_DEPENDS+= json-glib-1.0.0:${PORTSDIR}/devel/json-glib CONFIGURE_ARGS+= --enable-json --with-json=json-glib PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .elif defined(WITH_JSON_C) LIB_DEPENDS+= json.0:${PORTSDIR}/devel/json-c CONFIGURE_ARGS+= --enable-json --with-json=json-c PLIST_FILES+= lib/syslog-ng/libtfjson.la lib/syslog-ng/libtfjson.so .else CONFIGURE_ARGS+= --disable-json .endif So, in theory json support was disabled... And another backtrace, this time from openSUSE: (gdb) backtrace #0 0xb7684e8d in cfg_tree_compile_node () from /lib/libsyslog-ng-3.4.0alpha1.so #1 0xb7684f60 in cfg_tree_compile_node () from /lib/libsyslog-ng-3.4.0alpha1.so #2 0xb7685557 in cfg_tree_compile_rule () from /lib/libsyslog-ng-3.4.0alpha1.so #3 0xb7685954 in cfg_tree_compile () from /lib/libsyslog-ng-3.4.0alpha1.so #4 0xb76859ee in cfg_tree_start () from /lib/libsyslog-ng-3.4.0alpha1.so #5 0xb76808c0 in cfg_init () from /lib/libsyslog-ng-3.4.0alpha1.so #6 0xb76a07ea in main_loop_init () from /lib/libsyslog-ng-3.4.0alpha1.so #7 0x080491a7 in main () (gdb) Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

3 11

afsmtp packaging problems
by Peter Czanik 23 Mar '12

23 Mar '12

Hello, There are a couple of problem with smtp support. Tested on openSUSE and FreeBSD. Neither have libesmtp installed, at least not as default. So configure is right: SMTP support (module) : no On the other hand: - libafsmtp is built and installed - unlike the others, it is installed as a versioned library: linux-v3g3:~ # ls -l /lib/syslog-ng/ total 556 -rwxr-xr-x 1 root root 73212 Mar 21 09:14 libaffile.so -rwxr-xr-x 1 root root 48567 Mar 21 09:14 libafprog.so lrwxrwxrwx 1 root root 18 Mar 21 09:18 libafsmtp.so -> libafsmtp.so.0.0.0 lrwxrwxrwx 1 root root 18 Mar 21 09:18 libafsmtp.so.0 -> libafsmtp.so.0.0.0 -rwxr-xr-x 1 root root 6445 Mar 21 09:14 libafsmtp.so.0.0.0 -rwxr-xr-x 1 root root 118106 Mar 21 09:14 libafsocket.so -rwxr-xr-x 1 root root 34040 Mar 21 09:14 libafuser.so -rwxr-xr-x 1 root root 23092 Mar 21 09:14 libbasicfuncs.so -rwxr-xr-x 1 root root 11819 Mar 21 09:14 libconfgen.so -rwxr-xr-x 1 root root 38464 Mar 21 09:14 libcsvparser.so -rwxr-xr-x 1 root root 105045 Mar 21 09:14 libdbparser.so -rwxr-xr-x 1 root root 33668 Mar 21 09:14 libdummy.so -rwxr-xr-x 1 root root 21134 Mar 21 09:14 libsyslogformat.so -rwxr-xr-x 1 root root 28448 Mar 21 09:14 libsyslog-ng-crypto.so -rwxr-xr-x 1 root root 7339 Mar 21 09:14 libtfuuid.so ToDo: - option to disable smtp support - do not build/install it if build dependency is unavailable - do not install it as a versioned library Bye, -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 1

Syslog-NG forwarding the messages to a Remote TCP port with more than one message in a single packet
by anji prassana 18 Mar '12

18 Mar '12

I am using syslog-ng *version * [root@Cypher-210 ~]# /usr/local/sbin/syslog-ng -V syslog-ng 3.3.3 Installer-Version: 3.3.3 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#d199a1980be6b23fe24189e86a882812288e292c Compile-Date: Mar 6 2012 13:06:17 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,parser,syslogformat Available-Modules: afprog,convertfuncs,affile,afsocket-tls,confgen,csvparser,syslogformat,dbparser,basicfuncs,afmongodb,dummy,afuser,afsocket Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: off Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on Problem: ------------ Syslog-ng is forwarding the TCP packets to a Remote TCP port with More than one message in a single TCP Packet.But, in between the messages inside a packet, It doesn't includes any delimiter like "\n" or some other.But, the receiving Program listens on TCP requires a newline as a delimiter b/w messages so as to parse the messages individually. But, As Syslog-ng doesn't includes a separator b/w messages in a packet, The Receiver simply considering whole messages in a packet as a single Message which leads to false and unacceptable parsing. Please look into the following messages exists in a single TCP packet.The one I colored red is a start of new message. root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shadow *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/head -n 4 /etc/inittab *<13>*mar 6 17:22:15 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/usr/bin/tail -n 1 *<13>*mar 6 17:22:15 cypher-210 sshd[5583]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:15 cypher-210 sshd[5765]: accepted password for root from 10.0.15.218 port 44258 ssh2 <13>mar 6 17:22:15 cypher-210 sshd[5765]: pam_unix(sshd:session): session opened for user root by (uid=0) *<13>*mar 6 17:22:16 cypher-210 sshd[5278]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 pam_timestamp_check: pam `/' permissions are lax <13>mar 6 17:22:17 cypher-210 sshd[5281]: pam_unix(sshd:session): session closed for user root *<13>*mar 6 17:22:17 cypher-210 sudo: root : tty=unknown ; pwd=/ ; user=root ; command=/bin/grep ^shutdown: /etc/shad These are the settings i made in the syslog-ng.conf file: source s_test { internal(); unix-stream("/dev/log"); udp(); file("/var/log/syslog-ng_local"); }; destination d_test { tcp("10.0.15.18" port(9500)); #My Program listening on tcp port 9500 requires newline as a separator for each message. }; log { source(s_test); destination(d_test); }; Can anyone kindly help me, 1. How can i limit one tcp packet to hold only one message? [or] 2. How can I add newline as a delimiter b/w each message before the packet(s) are forwarding to a destination so that it could parse properly? Your Help is really appreciate.. Thanks&Regards Anjaneyulu P

3 3

Packet fragmentation issue
by José Moreno 18 Mar '12

18 Mar '12

Hi all, I'm running syslog-ng 2.4.1, log sources send to a log server which beside keeping the original data as is in files, forwards them in real time to a SIEM, spoofing source IP. My problem comes after some logs are too long to fit in a single frame, log server fragments those packets when sending them to SIEM and spoofing is not performed for them. Enviado desde mi iPhone

4 5

[Bug 165] New: macro sequenceId is empty
by bugzilla＠bugzilla.balabit.com 18 Mar '12

18 Mar '12

https://bugzilla.balabit.com/show_bug.cgi?id=165 Summary: macro sequenceId is empty Product: syslog-ng Version: 3.4.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: whille(a)163.com Type of the Report: bug Estimated Hours: 0.0 i'm using syslog-ng both as client and server, the problem is $.SDATA.meta.sequenceId is empty in server. is it client or server reason? any configure modify for this tag? conf: ... #destination d_mydest { file("/var/log/mylog.log" template("${.SDATA.meta.sequenceId}____$MSG\n"));}; server version:3.3.4 client version:3.2.2 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

Losing to much remote sent logs
by Daniel Neubacher 18 Mar '12

18 Mar '12

Hello there, I've started playing around with syslog-ng 3.3.4 ose a few days ago but I'm still experiencing some trouble. First of all we want to use syslog-ng to send all of our logs via udp to a central syslog server. This includes of course syslogs, apache logs and custom generated applogs. These logs are generated from 400 clients and produces a minimum of 300 mio. log lines a day. The problem is really simple: I'm losing log lines :P Most of the time everything goes well but when the logs are peaking high 1-5% logs are getting lost. Last night the stats of the server and a client said 0 drops but when I counted the lines I found lost lines. The server has 24g ram & 8 cores and I can rule out a network problem for sure. So now to my questions, has anyone else an idea where I can tweak my cfg or where I have to look to find more clues? Is tcp the only way to get around it? I've attached my syslog server cfg. The so_rcvbuf buffer is the same size as the os net.core.rmem settings. And as described in the various balabit blog posts I played around with log_fetch_limit and flush_lines already. syslog-ng.conf: @version: 3.3 options { threaded(yes); owner("root"); group("root"); perm(0660); dir_owner("root"); dir_group("root"); dir_perm(0770); create_dirs(yes); stats_freq(600); stats_level(2); chain_hostnames(yes); normalize_hostnames(yes); check_hostname(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(60); log_msg_size(16384); log_fifo_size(100000); use_fqdn(yes); #disabled 4 debugging # flush_lines(200); }; source s_src { unix-dgram("/dev/log"); internal(); file("/proc/kmsg" program_override("kernel")); }; source s_net { udp( log_fetch_limit(400) so_rcvbuf(51200000) keep_hostname(yes) keep_timestamp(no) ip("10.8.4.10") port(514) ); tcp( so_rcvbuf(51200000) so_keepalive(yes) keep_hostname(no) keep_timestamp(no) ip("10.8.4.10") port(514) ); syslog(); }; filter f_syslog { not program(access.log) and not program(error.log) and not program(beetle.log) and not program(edge.log); }; filter f_apache { program(access.log) or program(error.log); }; filter f_applogs { program(beetle.log) or program(edge.log); }; template t_plain { template("$MSG\n"); template_escape(no); }; destination d_messages { file("/var/log/messages"); }; destination d_remote { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST"); }; destination d_apache { file("/log/apache/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); }; destination d_applogs { file("/log/applogs/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM" template(t_plain)); }; log { source(s_src); destination(d_messages); }; log { source(s_net); filter(f_syslog); destination(d_remote); }; log { source(s_net); filter(f_apache); destination(d_apache); }; log { source(s_net); filter(f_applogs); destination(d_applogs); }; Thanks Daniel Neubacher

5 11

using syslog-ng and parsing data from both Windows XP and Windows 7 machines
by Mary A Waddick 18 Mar '12

18 Mar '12

Hi, I am using nxlog to send data from both Windows XP and Windows 7 machines to a Unix machine using syslog-ng . My nxlog.conf files are configured to send im_mseventlog data for the Windows XP boxes and im_msvistalog data for the Windows 7 boxes. (See attached file: new nxlog.conf) Therefore I get slightly different data for each machine. The examples on your website don't show me how to parse out all of the im_mseventlog or im_msvistalog data from the different columns in msg. Can you help me with getting the data? My syslog-ng.conf looks like this. I was told that the eventlog data would have the columns that I included in my table, but I am unable to figure out how to pull that data out of the msg column using the provided macros. I have searched and searched, but have not found any examples. (See attached file: syslog-ng.conf) Thank you for your help. Mary Anne Waddick Raytheon Technical Services Senior Software Engineer II (317) 306-2691 (desk)

2 1

syslog-ng blocking application
by Ulrich David 16 Mar '12

16 Mar '12

Hi, I have an ISC-DHCP who is logging on a local facility on a chrooted system : #options { # chain_hostnames(no); # use_dns(no); # flush_lines(1); # stats_freq(3600); #}; #source dhcp_src { # unix-stream("/chroot/dhcp/dev/log" max-connections(256)); #}; My syslog-ng take logs from this source, filter on program and then put the logs on two destinations : #filter f_chroot_l2 { # program(dhcpd); #}; # #destination dhcp_log { file("/var/log/dhcpd.log" perm(0644)); }; #destination dhcp_sql { program("/usr/local/sbin/dhcp_log_to_mysql"); }; # #log { # source(dhcp_src); # filter(f_chroot_l2); # destination(dhcp_log); # destination(dhcp_sql); # flags(final ); #}; Everything goes well… until I block my program "dhcp_log_to_mysql" (I do it with a lock on the mysql table it uses or with a sleep in the script - for test). When "dhcp_log_to_mysql" is blocked, logs continues to be written on dhcp_log for some minutes/seconds then it stops. My chrooted dhcpd hangs on that too. I don't know why syslog-ng is not just dropping logs (continuing eating them on the source). I don't have flow-control enabled… and I find nothing which can explain the problem. I try to resize the buffer output with very big number (log_fifo_size(45000000)) but it changes nothing. Logs and dhcpd hangs at the same moment. What can I do? Regards David

3 4

delay in forwarding logs
by Ivanov, Konstantin 16 Mar '12

16 Mar '12

I have two servers running syslog-ng 2.0.9 on Dell R710's. I did some benchmarking with loggen [1] and was able to generate 41000 msg/sec locally. When I try to forward log messages from one server to the other I only get about 60 msg/sec. tcpdump shows that only that many are sent over the wire as well. I test this by writing to the log file from a script. Any idea what can be the problem? Here are both configs from the sending and receiving servers: ----------- Sending syslog-ng server -------------- options { log_fifo_size(120000); time_reopen (1); time_reap(1); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); group(adm); perm(0640); dir_perm(0755); keep_hostname (yes); log_fetch_limit(60000); }; source s_zxtm { file("/var/log/zxtm/access_log",follow_freq(1), flags(no-parse)); }; destination d_logrelay { tcp(somehost.com port(514)); }; log { source(s_zxtm); destination(d_logrelay); flags(flow-control); }; --------------- Receiving syslog-ng server ------------------ options { log_fifo_size(3000000); time_reopen (1); time_reap(1); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (yes); group(adm); perm(0640); dir_perm(0755); keep_hostname (yes); stats_freq(600); sync(0); }; source s_zxtm { tcp(max-connections(500), port(514)); }; destination d_zxtm { file("/var/log/zxtm/incoming/$YEAR$MONTH$DAY$HOUR-access_log.aggregated" owner("glassfish") perm(0644) template("$MESSAGE\n") template_escape(no)); }; log { source(s_zxtm); destination(d_zxtm); flags(flow-control); }; -------------- [1]. loggen -iS -r 10000000 -s 200 -I 60 localhost 514 average rate = 41053.50 msg/sec, count=410535

2 1