- syslog-ng - lists.balabit.hu

Log messages format difference between version 2.0.9 and 3.3.4
by Alptugay Değirmencioğlu 05 Dec '12

05 Dec '12

Hello all. We are using syslog-ng to transmit messages to a stream and an application then reads the messages from the stream and parses them. We have been using 2.0.9 normally. But when we updated to 3.3.4 the message format coming to the stream has changed. Btw we are using CentOS 5 with kernerl version 3.5.3 In 2.0.9 we had the following config: ----------- destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSG\n") template-escape(yes) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; were receiving these kinds of messages: 2012-12-05\\01110:40:13\\011ogm42\\011dansguardian[21839]: 2012.12.5-10:40:13\\011-\\01110.42.1.208\\011http://i.radikal.com.tr/90x70/2012/12/05/fft68_mf1227270.Jpeg\\011*EXCEPTION*Ayricalikli_kullanici_adina_sahipsiniz.\\011GET\\0112499\\0110\\011\\0111\\011200\\011-\\011\\011\\011-\\011 18 You can see that \\011 was a kind of delimeter to us. ----------- And in the new syslog-ng version we have the following config (with 4 back slash \\\\, $MSG changed to $MSGHDR$MSGONLY and template-escape changed from yes to no): destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n") template-escape(no) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (The first parts are okay. But the \\011 int the $MSGONLY have become \t): 2012-12-05\\01110:45:24\\011localhost\\011dansguardian[21135]: 2012.12.5-10:45:24\t-\t127.0.0.1\thttp://www.ipnedir.com\t*SCANNED*\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t 76 ------------ With this config: destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n") template-escape(no) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (The first parts are okay. But the \\011 int the $MSGONLY are lost) 2012-12-05\\01110:48:33\\011localhost\\011dansguardian[21135]: 2012.12.5-10:48:33-127.0.0.1http://www.ipnedir.comGET361601200text/html- 73 ----------------- With this config: destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n") template-escape(no) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (Now the output has become completely wrong) 2012-12-0501110:52:06011localhost011dansguardian[21135]: 2012.12.5-10:52:06\t-\t127.0.0.1\thttp://www.ipnedir.com\t\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t 89 --------------------- And finally with this config: destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n") template-escape(yes) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (Note that this config is the same config with the one we used in 2.0.9 except the $MSGHDR$MSGONLY part ) 2012-12-0501110:53:58011localhost011dansguardian[21135]: 2012.12.5-10:53:58-127.0.0.1http://www.ipnedir.comGET361601200text/html- 184 --------------------------- So what we want to accomplish is that we want the old output back with the new syslog :) How can we achieve this? Btw when restarting syslog-ng service we get the following messages. I don't know if it has anything to do with this. Restarting syslog-ng: Error opening plugin module; module='dbparser', error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='log_prefix', change='program_override' WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Shutting down syslog-ng: [ OK ] Starting syslog-ng: Error opening plugin module; module='dbparser', error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='log_prefix', change='program_override' WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='10000' [ OK ]

2 1

writing error to a mounted drive
by Frank Scalzo 03 Dec '12

03 Dec '12

: Error opening file for writing; filename='/var/log/hosts/XXXX/XXXXX', error='Permission denied (13)' I made the directory hosts inside /var/log/ and before I mounted the drive to the path /var/log/hosts files would write just fine but when I mounted the drive in that path I get the permissions denied error. running on redhat REHL 6and syslog-NG 3.2 OSE

2 1

[PATCH] [patterndb] fix a missing XML semantic check in PDBLoader
by balint.kovacs＠balabit.com 03 Dec '12

03 Dec '12

From: Balint Kovacs <blint(a)blint.hu> pattnerndb has failed to check if we are within a rule when trying to add tags and values, so a patterndb xml containing these elements outside of a rule element would cause syslog-ng (and pdbtool) to segfault Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/patterndb.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 004d6b7..d2349f6 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -1154,10 +1154,20 @@ pdb_loader_text(GMarkupParseContext *context, const gchar *text, gsize text_len, } else if (state->in_tag) { + if (!state->in_rule) + { + *error = g_error_new(1, 0, "Unexpected <tag> element, must be within a rule"); + return; + } pdb_message_add_tag(state->current_message, text); } else if (state->value_name) { + if (!state->in_rule) + { + *error = g_error_new(1, 0, "Unexpected <value> element, must be within a rule"); + return; + } if (!state->current_message->values) state->current_message->values = g_ptr_array_new(); -- 1.7.9.5

2 1

syslog uses wrong and weird hostnames
by Daniel Neubacher 28 Nov '12

28 Nov '12

Hello there, My syslog-ng is logging fine from 700 servers but a few times per day it is logging into a wrong folder which is created by the $HOST variable. Normally syslog should use a hostname like host-1.worker.foobar.com but a few times a day it creates a folder names like: 4.22 p6-allrouter? p6-localnet p6-mc? p6-mcastpref p6-mcastpref? host-1.worker.foobar.xcom and other weird names... It only logs one or two lines and then uses the right folder again. Does anyone have a clue where I have to look for the problem? Here are the option line from the server and the client: Client: @version: 3.3 options { threaded(yes); use_dns(yes); use_fqdn(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(10); log_msg_size(256000); log_fifo_size(100000); normalize_hostnames(yes); check_hostname(yes); bad_hostname("^gconfd$"); create_dirs(yes); owner("root"); group("root"); perm(0640); time_reopen(30); }; Server: options { threaded(yes); owner("root"); group("root"); perm(0660); dir_owner("root"); dir_group("root"); dir_perm(0770); create_dirs(yes); chain_hostnames(no); normalize_hostnames(yes); check_hostname(yes); keep_hostname(no); use_fqdn(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(60); log_msg_size(256000); log_fifo_size(1000000); }; Destination example: destination d_syslog { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"); }; -- Daniel Neubacher, Network Administrator daniel.neubacher(a)xing.com<mailto:daniel.neubacher@xing.com> XING AG Gaensemarkt 43, 20354 Hamburg, Germany Tel. +49 40 419131-28, Fax +49 40 419131-11 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr. Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil Sunderland This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.

2 3

SYSLOG-NG 3.4 exiting with error code 2
by Deepak Gaur 27 Nov '12

27 Nov '12

Hello, I have cross-compiled syslog-ng for ARM target but it is exiting without showing any error code (on target). I am attaching strace output (syslog.txt). If anyone can help me out # syslog-ng -V syslog-ng 3.4.0alpha3 Installer-Version: 3.4.0alpha3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#master#709ab1252920063e6d4d0d610ad5c1ee5ebdfb75 Compile-Date: Nov 27 2012 14:59:53 Available-Modules: syslogformat,afsocket-notls,afuser,basicfuncs,confgen,dbparser,tfuuid,afmongodb,afsocket-tls,afprog,afsocket,affile,csvparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: on STRACE OUTPUT ------------------------ strace syslog-ng execve("/sbin/syslog-ng", ["syslog-ng"], [/* 15 vars */]) = 0 brk(0) = 0x13000 uname({sys="Linux", node="192.168.119.78", ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001d000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=10371, ...}) = 0 mmap2(NULL, 10371, PROT_READ, MAP_PRIVATE, 3, 0) = 0x4001e000 close(3) = 0 open("/lib/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\3407\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1513316, ...}) = 0 mmap2(NULL, 496856, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40026000 mprotect(0x40093000, 28672, PROT_NONE) = 0 mmap2(0x4009a000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6c) = 0x4009a000 mmap2(0x4009f000, 1240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4009f000 close(3) = 0 open("/lib/libnsl.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0004/\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=98605, ...}) = 0 mmap2(NULL, 116464, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400a0000 mprotect(0x400b2000, 28672, PROT_NONE) = 0 mmap2(0x400b9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11) = 0x400b9000 mmap2(0x400bb000, 5872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x400bb000 close(3) = 0 open("/lib/libgmodule-2.0.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0@\r\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=18254, ...}) = 0 mmap2(NULL, 45268, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400bd000 mprotect(0x400c0000, 28672, PROT_NONE) = 0 mmap2(0x400c7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0x400c7000 close(3) = 0 open("/lib/libgthread-2.0.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0000\5\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=7368, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40021000 mmap2(NULL, 36912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400c9000 mprotect(0x400ca000, 28672, PROT_NONE) = 0 mmap2(0x400d1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x400d1000 close(3) = 0 open("/lib/librt.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\300\26\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=39970, ...}) = 0 mmap2(NULL, 57872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400d3000 mprotect(0x400d9000, 28672, PROT_NONE) = 0 mmap2(0x400e0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5) = 0x400e0000 close(3) = 0 open("/lib/libglib-2.0.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\270D\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1762078, ...}) = 0 mmap2(NULL, 1584268, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400e2000 mprotect(0x4025c000, 28672, PROT_NONE) = 0 mmap2(0x40263000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x179) = 0x40263000 close(3) = 0 open("/lib/libevtlog.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\300\20\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=23326, ...}) = 0 mmap2(NULL, 49472, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40265000 mprotect(0x40269000, 28672, PROT_NONE) = 0 mmap2(0x40270000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3) = 0x40270000 close(3) = 0 open("/lib/libpcre.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\250\16\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=177343, ...}) = 0 mmap2(NULL, 200880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40272000 mprotect(0x4029b000, 28672, PROT_NONE) = 0 mmap2(0x402a2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28) = 0x402a2000 close(3) = 0 open("/lib/libdl.so.2", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\34\t\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=15728, ...}) = 0 mmap2(NULL, 41136, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x402a4000 mprotect(0x402a6000, 28672, PROT_NONE) = 0 mmap2(0x402ad000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0x402ad000 close(3) = 0 open("/lib/libpthread.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\330B\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=126565, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40022000 mmap2(NULL, 123376, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x402af000 mprotect(0x402c3000, 28672, PROT_NONE) = 0 mmap2(0x402ca000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13) = 0x402ca000 mmap2(0x402cc000, 4592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x402cc000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\330Q\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1498068, ...}) = 0 mmap2(NULL, 1217048, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x402ce000 mprotect(0x403eb000, 28672, PROT_NONE) = 0 mmap2(0x403f2000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11c) = 0x403f2000 mmap2(0x403f5000, 8728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x403f5000 close(3) = 0 open("/lib/libiconv.so.2", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\230\22\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0644, st_size=1061246, ...}) = 0 mmap2(NULL, 1048764, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x403f8000 mprotect(0x404f0000, 28672, PROT_NONE) = 0 mmap2(0x404f7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf7) = 0x404f7000 close(3) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40023000 mmap2(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x404f9000 set_tls(0x404f94a0, 0x40025674, 0x404fb5c8, 0x40025048, 0x40025048) = 0 mprotect(0x404f7000, 4096, PROT_READ) = 0 mprotect(0x403f2000, 8192, PROT_READ) = 0 mprotect(0x402ca000, 4096, PROT_READ) = 0 mprotect(0x402ad000, 4096, PROT_READ) = 0 mprotect(0x402a2000, 4096, PROT_READ) = 0 mprotect(0x40270000, 4096, PROT_READ) = 0 mprotect(0x40263000, 4096, PROT_READ) = 0 mprotect(0x400e0000, 4096, PROT_READ) = 0 mprotect(0x400d1000, 4096, PROT_READ) = 0 mprotect(0x400c7000, 4096, PROT_READ) = 0 mprotect(0x400b9000, 4096, PROT_READ) = 0 mprotect(0x4009a000, 8192, PROT_READ) = 0 mprotect(0x11000, 4096, PROT_READ) = 0 mprotect(0x40024000, 4096, PROT_READ) = 0 munmap(0x4001e000, 10371) = 0 set_tid_address(0x404f9048) = 577 set_robust_list(0x404f9050, 12) = 0 rt_sigaction(SIGRTMIN, {0x402b31e8, [], SA_SIGINFO|0x4000000}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0x402b2cb8, [], SA_RESTART|SA_SIGINFO|0x4000000}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0 brk(0) = 0x13000 brk(0x34000) = 0x34000 gettimeofday({973374482, 838271}, NULL) = 0 open("/proc/stat", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001e000 read(3, "cpu 507 0 15055 364178 0 13 4 0"..., 1024) = 355 read(3, "", 1024) = 0 close(3) = 0 munmap(0x4001e000, 4096) = 0 open("/proc/stat", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001e000 read(3, "cpu 507 0 15055 364205 0 13 4 0"..., 1024) = 355 read(3, "", 1024) = 0 close(3) = 0 munmap(0x4001e000, 4096) = 0 ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B115200 opost isig icanon echo ...}) = 0 ioctl(0, TIOCNOTTY) = 0 setsid() = 577 setrlimit(RLIMIT_NOFILE, {rlim_cur=4*1024, rlim_max=4*1024}) = 0 pipe([3, 4]) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x404f9048) = 578 close(4) = 0 read(3, "2\n", 6) = 2 close(3) = 0 exit_group(2) = ? +++ exited with 2 +++ -------------------------------------------------------------------------------- What could be the issue? Deepak Gaur

2 4

Queue size in syslog-ng?
by Abhijeet R 23 Nov '12

23 Nov '12

Hi all, In my setup, I send logs from all the servers to a central syslog server. This syslog-ng server also has a destination on a different port which is picked up by logstash's input block. (consider it just a consumer to those messages). What if this consumer goes down? How big is the queue? Is that configurable? How do I make sure that none of the messages are missed for the time the consumer is down? -- Cheers, Abhijeet R http://blog.abhijeetr.com

7 7

merging stuff from merge-queue/3.4
by Balazs Scheidler 22 Nov '12

22 Nov '12

Hi, I have started merging the branch $subject, however I've found some patches I couldn't merge. system-source: Use follow-freq(1) on FreeBSD prior to 9.1 Are you sure time based polling would work here? I mean time based polling also checks if the size of the file has changed, as it was meant to be used only with regular files. I'm afraid that the inability to use kqueue on /dev/klog would prevent using kqueue completely, as the syslog-ng file source code requires either regular files or pollable devices. dbparser: support inheriting properties in a corellation action I merged this one, although I have some doubts. It uses log_msg_clone_cow(), which does have some tricky semantics, as it assumes that the cloned message will not change, once cloned. I was thinking about removing clone_cow() completely because of this tricky stuff, and whenever we need a clone we're about to change the LogMessage anyway, and delaying the actual change means that we have to do two malloc() calls, instead of one. Can you test whether changing the message that triggered the action with a rewrite rule, keeps the cloned one intact? This would need a parallel path to the db-parser(). dbparser: Make the context length available to action conditions I didn't pick this one. I was thinking about using a template function to return the number of entries in the current context. This patch changes the log message while it could keep it read-only, and even if I wanted to do this macro-like, I'd have created a real macro. Both template formatting and template functions receive the context as arguments anyway, so the required information is readily available. -- Bazsi

5 8

[PATCH 0/3] Type hinting
by Gergely Nagy 22 Nov '12

22 Nov '12

The patches that will follow this overview implement type-hinting support into LogTemplate and value-pairs(), then extend the mongodb destination and format-json to use these hints. In the end, we'll do things like: mongodb(value-pairs(scope("selected-macros", "nv-pairs") pair("DATE", datetime("$UNIXTIME")) pair("PID", int("$PID")) pair(".bool", boolean("true")))); And this will use the appropriate types when inserting into mongodb. The $(format-json) template function works similarly: template("$(format-json --scope selected-macros,nv-pairs DATE=datetime($UNIXTIME) PID=int($PID) .bool=boolean(TRUE))\n"); I tried to make the code as performant as possible, there are no extra allocations performed, and in case there is no casting to perform, it's just an extra integer comparsion. It can be tweaked further, if so need be, but for now, I'm happy with the results. Nevertheless, there are a couple of things I'm not entirely satisfied with (namely, abusing the ScratchBuffers to store the type hint on the first byte; or the way the value-pairs CLI interface parses type hints), and I will fix them in the coming days. While using this isn't the most convenient thing ever, and setting default types is not possible from within drivers (technically, it is, but it's awkward), it's a step in the right direction. Once nvpairs gains proper type support, that'll make things a whole lot sweeter! This - and any future updates - are available from my feature/3.4/templates/cast-typehints branch.

2 9

RFC: syslog-parser
by Balazs Scheidler 22 Nov '12

22 Nov '12

Hi, I've figured it is not nice from me that I keep pushing stuff to "master" without getting proper review from the list. So here's a attempt to change that. I've just pushed an experimental patch to the "syslog-parser" branch on github that allows the syslog parsing functionality to be used as a syslog-ng parser: https://github.com/balabit/syslog-ng-3.4/tree/syslog-parser Here's a short description on why this is useful (quoting the commit message): This patch creates a new parser, to explicitly parse messages as syslog messages. This can be used to selectively parse these messages based on some filters. Imagine this use-case: log { source { udp(port(2000) flags(no-parse)); }; parser { log { junction { log { filter { netmask("127.0.0.1/32"); }; parser { syslog-parser(); }; flags(final); }; log { filter { netmask("127.0.0.2/32"); }; parser { csv-parser(columns("C1", "C2", "C3")); }; flags(final); }; }; }; }; destination { file("/home/bazsi/logs/qqq" template("$(format-json --key *)\n")); }; }; Messages from 127.0.0.1 will get parsed as syslog messages, while 127.0.0.2 as a csv-parser() style messages. I'd welcome any kind of feedback, code or functionality wise. I still have some stuff to fix in this, but after initial feedback I'm going to merge it to master. Thanks in advance. -- Bazsi

6 8

Filter netmask not working as expected
by Agus 20 Nov '12

20 Nov '12

Hi guys, Just new, and created the following conf for testing purposes. The problem is that i get the logs in both destinations despite the filter. @version: 3.4 @include "scl.conf" options { keep_hostname(yes); normalize_hostnames(yes); threaded(yes); ts_format(iso); # Adds TZ #use_fqdn(yes); use_dns(no); }; source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; destination d_my_mac { file("/var/log/mymac"); }; filter f_my_mac { netmask(10.24.18.2/255.255.255.255); }; log { source(s_network); filter(f_my_mac); destination(d_my_mac); flags(final); }; log { source(s_local); # uncomment this line to open port 514 to receive messages source(s_network); #destination(d_central_udp); destination(d_local); }; as netmask i also tried cidr /24 and same thing. The problem is that i get the logs in both destinations. I only want to have them in my_mac Thanks!

2 2