- syslog-ng - lists.balabit.hu

syslog-ng OSE 3.4 Administrator Guide draft update
by Fekete Róbert 07 Dec '12

07 Dec '12

Hi, My colleague, Ancsur Sebestyen had some time this week to work on the 3.4 admin guide a bit. Still far from being finished or up-to-date, the following new stuff has been documented since the first draft: * The tfhash template function * The JSON parser * The --disable-json compile option * The @PCRE@ patterndb parser * Bug 204: dbparser docs not clear for @NUMBER@ * Bug 205 Docs for macro MSGHDR incorrect * The LOGHOST macro * The syslog-ng-ctl reload option is included in the syslog-ng-ctl man page Also, the links to the docs have changed. The new ones are: HTML: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… Single-page HTML: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… PDF: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… As always, any feedback is welcome. Robert Fekete

2 1

[PATCH] system(): Use follow-freq(1) on FreeBSD prior to 9.1
by Gergely Nagy 07 Dec '12

07 Dec '12

FreeBSD prior to 9.1 does not have a kqueue-pollable /dev/klog, so syslog-ng would fall back to using timers, except that follow-freq() is explicitly set to zero, also diabling timer-based polling. This has the effect of making syslog-ng not read from /dev/klog at all. To work around the issue, on FreeBSD prior to 9.1, use follow-freq(1) for /dev/klog when using the system() source. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- scl/system/generate-system-source.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/scl/system/generate-system-source.sh b/scl/system/generate-system-source.sh index 708a7a4..34cc009 100755 --- a/scl/system/generate-system-source.sh +++ b/scl/system/generate-system-source.sh @@ -74,7 +74,17 @@ EOF cat <<EOF unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); -file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse)); +EOF + case "$osversion" in + 7.*|8.*|9.0*) + ffreq=1 + ;; + *) + ffreq=0 + ;; + esac + cat <<EOF +file("/dev/klog" follow-freq($ffreq) program-override("kernel") flags(no-parse)); EOF ;; GNU/kFreeBSD) -- 1.7.10.4

2 1

Embedded log path gives syntax error
by Anton Koldaev 06 Dec '12

06 Dec '12

https://gist.github.com/40c233f40ebc8bc9163e -- Best regards, Koldaev Anton

2 1

openSUSE packages updated, SLES 11 SP2 added
by Peter Czanik 06 Dec '12

06 Dec '12

Hello, My openSUSE packages are now updated. There were two major changes: - binaries moved from / to /usr (supported by openSUSE 12.1+ and SLES 11 SP2) - lot of legacy ifdefs removed, so old releases ( <12.1, unsupported by the openSUSE project) are not supported any more This has two advantages: a lot cleaner spec file, and SLES support added. SLES now also has SSL support. The SLES package was tested only on SLES 11 SP2, but will most likely also work on older SLES 11 releases as long as /usr is not a separate partition. As usual, my 3.3 packages are available at https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 Once 3.4 beta is released, I'll rework my 3.4 packages too. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Which way is better/faster
by Anton Koldaev 06 Dec '12

06 Dec '12

Let's imagine we have one source with some logs: 80% - normal messages (all messages) 20% - specific messages (some complex regex) Normal messages filter includes specific messages. We need to put these messages into separate files. Which way will give more performance? 1. log(source(s),filter(non-specific),destination(normal),flags(final)) log(source(s),destination(specific),flags(final)) OR 2. log(source(s),filter(specific),destination(specific)) log(source(s),filter(normal),destination(normal),flags(final)) OR 3. log(source(s),filter(normal),destination(normal)) log(source(s),filter(specific),destination(specific),flags(final)) -- Best regards, Koldaev Anton

2 1

Embedded log statements performance
by Anton Koldaev 05 Dec '12

05 Dec '12

Which way is faster: 1. log { source(s_localhost); message("example1"); destination(d_file1); flags(final); log {message("example2"); destination(d_file2); }; }; OR 2. log { source(s_localhost); message("example1"); destination(d_file1); flags(final);}; log { source(s_localhost); message("example2"); destination(d_file2); flags(final);}; -- Best regards, Koldaev Anton

1 0

Log messages format difference between version 2.0.9 and 3.3.4
by Alptugay Değirmencioğlu 05 Dec '12

05 Dec '12

Hello all. We are using syslog-ng to transmit messages to a stream and an application then reads the messages from the stream and parses them. We have been using 2.0.9 normally. But when we updated to 3.3.4 the message format coming to the stream has changed. Btw we are using CentOS 5 with kernerl version 3.5.3 In 2.0.9 we had the following config: ----------- destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSG\n") template-escape(yes) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; were receiving these kinds of messages: 2012-12-05\\01110:40:13\\011ogm42\\011dansguardian[21839]: 2012.12.5-10:40:13\\011-\\01110.42.1.208\\011http://i.radikal.com.tr/90x70/2012/12/05/fft68_mf1227270.Jpeg\\011*EXCEPTION*Ayricalikli_kullanici_adina_sahipsiniz.\\011GET\\0112499\\0110\\011\\0111\\011200\\011-\\011\\011\\011-\\011 18 You can see that \\011 was a kind of delimeter to us. ----------- And in the new syslog-ng version we have the following config (with 4 back slash \\\\, $MSG changed to $MSGHDR$MSGONLY and template-escape changed from yes to no): destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n") template-escape(no) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (The first parts are okay. But the \\011 int the $MSGONLY have become \t): 2012-12-05\\01110:45:24\\011localhost\\011dansguardian[21135]: 2012.12.5-10:45:24\t-\t127.0.0.1\thttp://www.ipnedir.com\t*SCANNED*\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t 76 ------------ With this config: destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n") template-escape(no) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (The first parts are okay. But the \\011 int the $MSGONLY are lost) 2012-12-05\\01110:48:33\\011localhost\\011dansguardian[21135]: 2012.12.5-10:48:33-127.0.0.1http://www.ipnedir.comGET361601200text/html- 73 ----------------- With this config: destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n") template-escape(no) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (Now the output has become completely wrong) 2012-12-0501110:52:06011localhost011dansguardian[21135]: 2012.12.5-10:52:06\t-\t127.0.0.1\thttp://www.ipnedir.com\t\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t 89 --------------------- And finally with this config: destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n") template-escape(yes) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (Note that this config is the same config with the one we used in 2.0.9 except the $MSGHDR$MSGONLY part ) 2012-12-0501110:53:58011localhost011dansguardian[21135]: 2012.12.5-10:53:58-127.0.0.1http://www.ipnedir.comGET361601200text/html- 184 --------------------------- So what we want to accomplish is that we want the old output back with the new syslog :) How can we achieve this? Btw when restarting syslog-ng service we get the following messages. I don't know if it has anything to do with this. Restarting syslog-ng: Error opening plugin module; module='dbparser', error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='log_prefix', change='program_override' WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Shutting down syslog-ng: [ OK ] Starting syslog-ng: Error opening plugin module; module='dbparser', error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='log_prefix', change='program_override' WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='10000' [ OK ]

2 1

writing error to a mounted drive
by Frank Scalzo 03 Dec '12

03 Dec '12

: Error opening file for writing; filename='/var/log/hosts/XXXX/XXXXX', error='Permission denied (13)' I made the directory hosts inside /var/log/ and before I mounted the drive to the path /var/log/hosts files would write just fine but when I mounted the drive in that path I get the permissions denied error. running on redhat REHL 6and syslog-NG 3.2 OSE

2 1

[PATCH] [patterndb] fix a missing XML semantic check in PDBLoader
by balint.kovacs＠balabit.com 03 Dec '12

03 Dec '12

From: Balint Kovacs <blint(a)blint.hu> pattnerndb has failed to check if we are within a rule when trying to add tags and values, so a patterndb xml containing these elements outside of a rule element would cause syslog-ng (and pdbtool) to segfault Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/patterndb.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 004d6b7..d2349f6 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -1154,10 +1154,20 @@ pdb_loader_text(GMarkupParseContext *context, const gchar *text, gsize text_len, } else if (state->in_tag) { + if (!state->in_rule) + { + *error = g_error_new(1, 0, "Unexpected <tag> element, must be within a rule"); + return; + } pdb_message_add_tag(state->current_message, text); } else if (state->value_name) { + if (!state->in_rule) + { + *error = g_error_new(1, 0, "Unexpected <value> element, must be within a rule"); + return; + } if (!state->current_message->values) state->current_message->values = g_ptr_array_new(); -- 1.7.9.5

2 1

syslog uses wrong and weird hostnames
by Daniel Neubacher 28 Nov '12

28 Nov '12

Hello there, My syslog-ng is logging fine from 700 servers but a few times per day it is logging into a wrong folder which is created by the $HOST variable. Normally syslog should use a hostname like host-1.worker.foobar.com but a few times a day it creates a folder names like: 4.22 p6-allrouter? p6-localnet p6-mc? p6-mcastpref p6-mcastpref? host-1.worker.foobar.xcom and other weird names... It only logs one or two lines and then uses the right folder again. Does anyone have a clue where I have to look for the problem? Here are the option line from the server and the client: Client: @version: 3.3 options { threaded(yes); use_dns(yes); use_fqdn(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(10); log_msg_size(256000); log_fifo_size(100000); normalize_hostnames(yes); check_hostname(yes); bad_hostname("^gconfd$"); create_dirs(yes); owner("root"); group("root"); perm(0640); time_reopen(30); }; Server: options { threaded(yes); owner("root"); group("root"); perm(0660); dir_owner("root"); dir_group("root"); dir_perm(0770); create_dirs(yes); chain_hostnames(no); normalize_hostnames(yes); check_hostname(yes); keep_hostname(no); use_fqdn(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(60); log_msg_size(256000); log_fifo_size(1000000); }; Destination example: destination d_syslog { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"); }; -- Daniel Neubacher, Network Administrator daniel.neubacher(a)xing.com<mailto:daniel.neubacher@xing.com> XING AG Gaensemarkt 43, 20354 Hamburg, Germany Tel. +49 40 419131-28, Fax +49 40 419131-11 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr. Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil Sunderland This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.

2 3

SYSLOG-NG 3.4 exiting with error code 2
by Deepak Gaur 27 Nov '12

27 Nov '12

Hello, I have cross-compiled syslog-ng for ARM target but it is exiting without showing any error code (on target). I am attaching strace output (syslog.txt). If anyone can help me out # syslog-ng -V syslog-ng 3.4.0alpha3 Installer-Version: 3.4.0alpha3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#master#709ab1252920063e6d4d0d610ad5c1ee5ebdfb75 Compile-Date: Nov 27 2012 14:59:53 Available-Modules: syslogformat,afsocket-notls,afuser,basicfuncs,confgen,dbparser,tfuuid,afmongodb,afsocket-tls,afprog,afsocket,affile,csvparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: on STRACE OUTPUT ------------------------ strace syslog-ng execve("/sbin/syslog-ng", ["syslog-ng"], [/* 15 vars */]) = 0 brk(0) = 0x13000 uname({sys="Linux", node="192.168.119.78", ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001d000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=10371, ...}) = 0 mmap2(NULL, 10371, PROT_READ, MAP_PRIVATE, 3, 0) = 0x4001e000 close(3) = 0 open("/lib/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\3407\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1513316, ...}) = 0 mmap2(NULL, 496856, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40026000 mprotect(0x40093000, 28672, PROT_NONE) = 0 mmap2(0x4009a000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6c) = 0x4009a000 mmap2(0x4009f000, 1240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4009f000 close(3) = 0 open("/lib/libnsl.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0004/\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=98605, ...}) = 0 mmap2(NULL, 116464, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400a0000 mprotect(0x400b2000, 28672, PROT_NONE) = 0 mmap2(0x400b9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11) = 0x400b9000 mmap2(0x400bb000, 5872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x400bb000 close(3) = 0 open("/lib/libgmodule-2.0.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0@\r\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=18254, ...}) = 0 mmap2(NULL, 45268, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400bd000 mprotect(0x400c0000, 28672, PROT_NONE) = 0 mmap2(0x400c7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0x400c7000 close(3) = 0 open("/lib/libgthread-2.0.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0000\5\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=7368, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40021000 mmap2(NULL, 36912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400c9000 mprotect(0x400ca000, 28672, PROT_NONE) = 0 mmap2(0x400d1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x400d1000 close(3) = 0 open("/lib/librt.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\300\26\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=39970, ...}) = 0 mmap2(NULL, 57872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400d3000 mprotect(0x400d9000, 28672, PROT_NONE) = 0 mmap2(0x400e0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5) = 0x400e0000 close(3) = 0 open("/lib/libglib-2.0.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\270D\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1762078, ...}) = 0 mmap2(NULL, 1584268, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400e2000 mprotect(0x4025c000, 28672, PROT_NONE) = 0 mmap2(0x40263000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x179) = 0x40263000 close(3) = 0 open("/lib/libevtlog.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\300\20\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=23326, ...}) = 0 mmap2(NULL, 49472, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40265000 mprotect(0x40269000, 28672, PROT_NONE) = 0 mmap2(0x40270000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3) = 0x40270000 close(3) = 0 open("/lib/libpcre.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\250\16\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=177343, ...}) = 0 mmap2(NULL, 200880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40272000 mprotect(0x4029b000, 28672, PROT_NONE) = 0 mmap2(0x402a2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28) = 0x402a2000 close(3) = 0 open("/lib/libdl.so.2", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\34\t\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=15728, ...}) = 0 mmap2(NULL, 41136, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x402a4000 mprotect(0x402a6000, 28672, PROT_NONE) = 0 mmap2(0x402ad000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0x402ad000 close(3) = 0 open("/lib/libpthread.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\330B\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=126565, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40022000 mmap2(NULL, 123376, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x402af000 mprotect(0x402c3000, 28672, PROT_NONE) = 0 mmap2(0x402ca000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13) = 0x402ca000 mmap2(0x402cc000, 4592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x402cc000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\330Q\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1498068, ...}) = 0 mmap2(NULL, 1217048, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x402ce000 mprotect(0x403eb000, 28672, PROT_NONE) = 0 mmap2(0x403f2000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11c) = 0x403f2000 mmap2(0x403f5000, 8728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x403f5000 close(3) = 0 open("/lib/libiconv.so.2", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\230\22\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0644, st_size=1061246, ...}) = 0 mmap2(NULL, 1048764, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x403f8000 mprotect(0x404f0000, 28672, PROT_NONE) = 0 mmap2(0x404f7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf7) = 0x404f7000 close(3) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40023000 mmap2(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x404f9000 set_tls(0x404f94a0, 0x40025674, 0x404fb5c8, 0x40025048, 0x40025048) = 0 mprotect(0x404f7000, 4096, PROT_READ) = 0 mprotect(0x403f2000, 8192, PROT_READ) = 0 mprotect(0x402ca000, 4096, PROT_READ) = 0 mprotect(0x402ad000, 4096, PROT_READ) = 0 mprotect(0x402a2000, 4096, PROT_READ) = 0 mprotect(0x40270000, 4096, PROT_READ) = 0 mprotect(0x40263000, 4096, PROT_READ) = 0 mprotect(0x400e0000, 4096, PROT_READ) = 0 mprotect(0x400d1000, 4096, PROT_READ) = 0 mprotect(0x400c7000, 4096, PROT_READ) = 0 mprotect(0x400b9000, 4096, PROT_READ) = 0 mprotect(0x4009a000, 8192, PROT_READ) = 0 mprotect(0x11000, 4096, PROT_READ) = 0 mprotect(0x40024000, 4096, PROT_READ) = 0 munmap(0x4001e000, 10371) = 0 set_tid_address(0x404f9048) = 577 set_robust_list(0x404f9050, 12) = 0 rt_sigaction(SIGRTMIN, {0x402b31e8, [], SA_SIGINFO|0x4000000}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0x402b2cb8, [], SA_RESTART|SA_SIGINFO|0x4000000}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0 brk(0) = 0x13000 brk(0x34000) = 0x34000 gettimeofday({973374482, 838271}, NULL) = 0 open("/proc/stat", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001e000 read(3, "cpu 507 0 15055 364178 0 13 4 0"..., 1024) = 355 read(3, "", 1024) = 0 close(3) = 0 munmap(0x4001e000, 4096) = 0 open("/proc/stat", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001e000 read(3, "cpu 507 0 15055 364205 0 13 4 0"..., 1024) = 355 read(3, "", 1024) = 0 close(3) = 0 munmap(0x4001e000, 4096) = 0 ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B115200 opost isig icanon echo ...}) = 0 ioctl(0, TIOCNOTTY) = 0 setsid() = 577 setrlimit(RLIMIT_NOFILE, {rlim_cur=4*1024, rlim_max=4*1024}) = 0 pipe([3, 4]) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x404f9048) = 578 close(4) = 0 read(3, "2\n", 6) = 2 close(3) = 0 exit_group(2) = ? +++ exited with 2 +++ -------------------------------------------------------------------------------- What could be the issue? Deepak Gaur

2 4

Queue size in syslog-ng?
by Abhijeet R 23 Nov '12

23 Nov '12

Hi all, In my setup, I send logs from all the servers to a central syslog server. This syslog-ng server also has a destination on a different port which is picked up by logstash's input block. (consider it just a consumer to those messages). What if this consumer goes down? How big is the queue? Is that configurable? How do I make sure that none of the messages are missed for the time the consumer is down? -- Cheers, Abhijeet R http://blog.abhijeetr.com

7 7

merging stuff from merge-queue/3.4
by Balazs Scheidler 22 Nov '12

22 Nov '12

Hi, I have started merging the branch $subject, however I've found some patches I couldn't merge. system-source: Use follow-freq(1) on FreeBSD prior to 9.1 Are you sure time based polling would work here? I mean time based polling also checks if the size of the file has changed, as it was meant to be used only with regular files. I'm afraid that the inability to use kqueue on /dev/klog would prevent using kqueue completely, as the syslog-ng file source code requires either regular files or pollable devices. dbparser: support inheriting properties in a corellation action I merged this one, although I have some doubts. It uses log_msg_clone_cow(), which does have some tricky semantics, as it assumes that the cloned message will not change, once cloned. I was thinking about removing clone_cow() completely because of this tricky stuff, and whenever we need a clone we're about to change the LogMessage anyway, and delaying the actual change means that we have to do two malloc() calls, instead of one. Can you test whether changing the message that triggered the action with a rewrite rule, keeps the cloned one intact? This would need a parallel path to the db-parser(). dbparser: Make the context length available to action conditions I didn't pick this one. I was thinking about using a template function to return the number of entries in the current context. This patch changes the log message while it could keep it read-only, and even if I wanted to do this macro-like, I'd have created a real macro. Both template formatting and template functions receive the context as arguments anyway, so the required information is readily available. -- Bazsi

5 8

[PATCH 0/3] Type hinting
by Gergely Nagy 22 Nov '12

22 Nov '12

The patches that will follow this overview implement type-hinting support into LogTemplate and value-pairs(), then extend the mongodb destination and format-json to use these hints. In the end, we'll do things like: mongodb(value-pairs(scope("selected-macros", "nv-pairs") pair("DATE", datetime("$UNIXTIME")) pair("PID", int("$PID")) pair(".bool", boolean("true")))); And this will use the appropriate types when inserting into mongodb. The $(format-json) template function works similarly: template("$(format-json --scope selected-macros,nv-pairs DATE=datetime($UNIXTIME) PID=int($PID) .bool=boolean(TRUE))\n"); I tried to make the code as performant as possible, there are no extra allocations performed, and in case there is no casting to perform, it's just an extra integer comparsion. It can be tweaked further, if so need be, but for now, I'm happy with the results. Nevertheless, there are a couple of things I'm not entirely satisfied with (namely, abusing the ScratchBuffers to store the type hint on the first byte; or the way the value-pairs CLI interface parses type hints), and I will fix them in the coming days. While using this isn't the most convenient thing ever, and setting default types is not possible from within drivers (technically, it is, but it's awkward), it's a step in the right direction. Once nvpairs gains proper type support, that'll make things a whole lot sweeter! This - and any future updates - are available from my feature/3.4/templates/cast-typehints branch.

2 9

RFC: syslog-parser
by Balazs Scheidler 22 Nov '12

22 Nov '12

Hi, I've figured it is not nice from me that I keep pushing stuff to "master" without getting proper review from the list. So here's a attempt to change that. I've just pushed an experimental patch to the "syslog-parser" branch on github that allows the syslog parsing functionality to be used as a syslog-ng parser: https://github.com/balabit/syslog-ng-3.4/tree/syslog-parser Here's a short description on why this is useful (quoting the commit message): This patch creates a new parser, to explicitly parse messages as syslog messages. This can be used to selectively parse these messages based on some filters. Imagine this use-case: log { source { udp(port(2000) flags(no-parse)); }; parser { log { junction { log { filter { netmask("127.0.0.1/32"); }; parser { syslog-parser(); }; flags(final); }; log { filter { netmask("127.0.0.2/32"); }; parser { csv-parser(columns("C1", "C2", "C3")); }; flags(final); }; }; }; }; destination { file("/home/bazsi/logs/qqq" template("$(format-json --key *)\n")); }; }; Messages from 127.0.0.1 will get parsed as syslog messages, while 127.0.0.2 as a csv-parser() style messages. I'd welcome any kind of feedback, code or functionality wise. I still have some stuff to fix in this, but after initial feedback I'm going to merge it to master. Thanks in advance. -- Bazsi

6 8

Filter netmask not working as expected
by Agus 20 Nov '12

20 Nov '12

Hi guys, Just new, and created the following conf for testing purposes. The problem is that i get the logs in both destinations despite the filter. @version: 3.4 @include "scl.conf" options { keep_hostname(yes); normalize_hostnames(yes); threaded(yes); ts_format(iso); # Adds TZ #use_fqdn(yes); use_dns(no); }; source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; destination d_my_mac { file("/var/log/mymac"); }; filter f_my_mac { netmask(10.24.18.2/255.255.255.255); }; log { source(s_network); filter(f_my_mac); destination(d_my_mac); flags(final); }; log { source(s_local); # uncomment this line to open port 514 to receive messages source(s_network); #destination(d_central_udp); destination(d_local); }; as netmask i also tried cidr /24 and same thing. The problem is that i get the logs in both destinations. I only want to have them in my_mac Thanks!

2 2

[PATCH 1/2] [dbparser] inherit nvpairs from parent in correlation action
by balint.kovacs＠balabit.com 16 Nov '12

16 Nov '12

From: Balint Kovacs <blint(a)blint.hu> This patch implements an inherit-name-value-pairs attribute for message type actions is patterndb correlation. In case this is set to TRUE, the triggering log message is cloned and all name-value pairs are automagically copied to the emitted message. The values set in the values XML subtree overwrite the original values. In case it is set to FALSE or unspecified, a new message is generated and only the values set manually are added to it. <actions> <action> <message inherit-name-value-pairs='TRUE'> <values> <value name="PROGRAM">my-program-name</value> <value name="MSG">Hello new message!</value> <value name="contextid">${session-id}@2</value> </values> </message> </action> </actions> Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/patterndb-int.h | 1 + modules/dbparser/patterndb.c | 32 +++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/modules/dbparser/patterndb-int.h b/modules/dbparser/patterndb-int.h index 0434847..154b686 100644 --- a/modules/dbparser/patterndb-int.h +++ b/modules/dbparser/patterndb-int.h @@ -126,6 +126,7 @@ typedef struct _PDBAction guint8 content_type; guint16 rate; guint32 id:8, rate_quantum:24; + gboolean inherit_nvpairs; union { PDBMessage message; diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 8720ea0..168a24c 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -388,6 +388,17 @@ pdb_action_set_trigger(PDBAction *self, const gchar *trigger, GError **error) g_set_error(error, 0, 1, "Unknown trigger type: %s", trigger); } +void +pdb_action_set_inheritance(PDBAction *self, const gchar *inherit_nvpairs, GError **error) +{ + if (strcmp(inherit_nvpairs, "TRUE") == 0) + self->inherit_nvpairs = TRUE; + else if (strcmp(inherit_nvpairs, "FALSE") == 0) + self->inherit_nvpairs = FALSE; + else + g_set_error(error, 0, 1, "Unknown inheritance type: %s", inherit_nvpairs); +} + PDBAction * pdb_action_new(gint id) { @@ -398,6 +409,7 @@ pdb_action_new(gint id) self->trigger = RAT_MATCH; self->content_type = RAC_NONE; self->id = id; + self->inherit_nvpairs = FALSE; return self; } @@ -552,9 +564,18 @@ pdb_rule_run_actions(PDBRule *self, gint trigger, PatternDB *db, PDBContext *con case RAC_NONE: break; case RAC_MESSAGE: - genmsg = log_msg_new_empty(); - genmsg->flags |= LF_LOCAL; - genmsg->timestamps[LM_TS_STAMP] = msg->timestamps[LM_TS_STAMP]; + if (action->inherit_nvpairs) + { + LogPathOptions path_options = LOG_PATH_OPTIONS_INIT; + path_options.ack_needed = FALSE; + genmsg = log_msg_clone_cow(msg, &path_options); + } + else + { + genmsg = log_msg_new_empty(); + genmsg->flags |= LF_LOCAL; + genmsg->timestamps[LM_TS_STAMP] = msg->timestamps[LM_TS_STAMP]; + } if (context) { switch (context->key.scope) @@ -942,6 +963,11 @@ pdb_loader_start_element(GMarkupParseContext *context, const gchar *element_name } else if (strcmp(element_name, "message") == 0) { + for (i = 0; attribute_names[i]; i++) + { + if (strcmp(attribute_names[i], "inherit-name-value-pairs") == 0) + pdb_action_set_inheritance(state->current_action, attribute_values[i], error); + } if (!state->in_action) { *error = g_error_new(1, 0, "Unexpected <message> element, it must be inside an action"); -- 1.7.9.5

5 9

[PATCH (3.4)] afmongodb: Add support for datetime fields
by Gergely Nagy 15 Nov '12

15 Nov '12

For TTL collections primarly (but useful for other scenarios aswell), introduce a new option: datetime(FIELD_NAME). This allows one to mark a single field anywhere in the structure as a datetime field, and the driver will attempt to convert it to a proper datetime, by assuming it is a unix timestamp. If the conversion fails, or the field is not found, it will not be inserted. Usage: mongodb(datetime("some.date") value-pairs( key("MESSAGE") pair("some.date", "$UNIXTIME"))) This will result in something like: { "_id" : ObjectId("509b8ffc1a1006213a000001"), "some" : { "date" : ISODate("2012-11-08T10:56:59Z") }, "MESSAGE" : "foobar" } Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb-grammar.ym | 2 ++ modules/afmongodb/afmongodb-parser.c | 1 + modules/afmongodb/afmongodb.c | 61 +++++++++++++++++++++++++++++--- modules/afmongodb/afmongodb.h | 1 + 4 files changed, 61 insertions(+), 4 deletions(-) diff --git a/modules/afmongodb/afmongodb-grammar.ym b/modules/afmongodb/afmongodb-grammar.ym index f108f26..40eccc0 100644 --- a/modules/afmongodb/afmongodb-grammar.ym +++ b/modules/afmongodb/afmongodb-grammar.ym @@ -53,6 +53,7 @@ extern ValuePairsTransformSet *last_vp_transset; %token KW_SERVERS %token KW_SAFE_MODE %token KW_PATH +%token KW_DATETIME %% @@ -97,6 +98,7 @@ afmongodb_option | KW_USERNAME '(' string ')' { afmongodb_dd_set_user(last_driver, $3); free($3); } | KW_PASSWORD '(' string ')' { afmongodb_dd_set_password(last_driver, $3); free($3); } | KW_SAFE_MODE '(' yesno ')' { afmongodb_dd_set_safe_mode(last_driver, $3); } + | KW_DATETIME '(' string ')' { afmongodb_dd_set_datetime(last_driver, $3); free($3); } | value_pair_option { afmongodb_dd_set_value_pairs(last_driver, $1); } | dest_driver_option ; diff --git a/modules/afmongodb/afmongodb-parser.c b/modules/afmongodb/afmongodb-parser.c index 8aa55c1..1c51154 100644 --- a/modules/afmongodb/afmongodb-parser.c +++ b/modules/afmongodb/afmongodb-parser.c @@ -39,6 +39,7 @@ static CfgLexerKeyword afmongodb_keywords[] = { { "host", KW_HOST }, { "port", KW_PORT }, { "path", KW_PATH }, + { "datetime", KW_DATETIME }, { NULL } }; diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 205ab98..bb267a5 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -22,6 +22,7 @@ */ #include <time.h> +#include <stdlib.h> #include "afmongodb.h" #include "afmongodb-parser.h" @@ -32,6 +33,7 @@ #include "nvtable.h" #include "logqueue.h" #include "value-pairs.h" +#include "scratch-buffers.h" #include "mongo.h" @@ -55,6 +57,7 @@ typedef struct gint port; gboolean safe_mode; + gchar *datetime_field; gchar *user; gchar *password; @@ -207,6 +210,15 @@ afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state) self->safe_mode = state; } +void +afmongodb_dd_set_datetime(LogDriver *d, const gchar *field) +{ + MongoDBDestDriver *self = (MongoDBDestDriver *)d; + + g_free(self->datetime_field); + self->datetime_field = g_strdup(field); +} + /* * Utilities */ @@ -343,7 +355,7 @@ afmongodb_vp_obj_end(const gchar *name, if (prev_data) root = (bson *)*prev_data; else - root = (bson *)user_data; + root = (bson *)(((gpointer *)user_data)[0]); if (prefix_data) { @@ -357,6 +369,30 @@ afmongodb_vp_obj_end(const gchar *name, } static gboolean +afmongodb_field_is_datetime(const gchar *prefix, const gchar *name, + const gchar *datetime_field) +{ + size_t plen = 0; + + if (!datetime_field) + return FALSE; + + if (prefix) + plen = strlen(prefix); + + if (strncmp(datetime_field, prefix, plen) != 0) + return FALSE; + + if (datetime_field[plen] != '.') + return FALSE; + + if (strcmp(&datetime_field[plen + 1], name) != 0) + return FALSE; + + return TRUE; +} + +static gboolean afmongodb_vp_process_value(const gchar *name, const gchar *prefix, const gchar *value, gpointer *prefix_data, gpointer user_data) @@ -366,7 +402,7 @@ afmongodb_vp_process_value(const gchar *name, const gchar *prefix, if (prefix_data) o = (bson *)*prefix_data; else - o = (bson *)user_data; + o = (bson *)(((gpointer *)user_data)[0]); if (name[0] == '.') { @@ -378,7 +414,20 @@ afmongodb_vp_process_value(const gchar *name, const gchar *prefix, bson_append_string (o, tx_name, value, -1); } else - bson_append_string (o, name, value, -1); + { + MongoDBDestDriver *self = (MongoDBDestDriver *)(((gpointer *)user_data)[1]); + + if (afmongodb_field_is_datetime(prefix, name, self->datetime_field)) + { + gchar *endptr; + gint64 datetime = (gint64)strtoul(value, &endptr, 10) * 1000; + + if (endptr[0] == '\0') + bson_append_utc_datetime(o, name, datetime); + } + else + bson_append_string (o, name, value, -1); + } return FALSE; } @@ -390,6 +439,7 @@ afmongodb_worker_insert (MongoDBDestDriver *self) guint8 *oid; LogMessage *msg; LogPathOptions path_options = LOG_PATH_OPTIONS_INIT; + gpointer user_data[] = { NULL, self }; afmongodb_dd_connect(self, TRUE); @@ -408,11 +458,13 @@ afmongodb_worker_insert (MongoDBDestDriver *self) bson_append_oid (self->bson, "_id", oid); g_free (oid); + user_data[0] = self->bson; + value_pairs_walk(self->vp, afmongodb_vp_obj_start, afmongodb_vp_process_value, afmongodb_vp_obj_end, - msg, self->seq_num, self->bson); + msg, self->seq_num, user_data); bson_finish (self->bson); if (!mongo_sync_cmd_insert_n(self->conn, self->ns, 1, @@ -650,6 +702,7 @@ afmongodb_dd_free(LogPipe *d) string_list_free(self->servers); if (self->vp) value_pairs_free(self->vp); + g_free(self->datetime_field); log_dest_driver_free(d); } diff --git a/modules/afmongodb/afmongodb.h b/modules/afmongodb/afmongodb.h index 6a61616..76f8b83 100644 --- a/modules/afmongodb/afmongodb.h +++ b/modules/afmongodb/afmongodb.h @@ -39,6 +39,7 @@ void afmongodb_dd_set_password(LogDriver *d, const gchar *password); void afmongodb_dd_set_value_pairs(LogDriver *d, ValuePairs *vp); void afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state); void afmongodb_dd_set_path(LogDriver *d, const gchar *path); +void afmongodb_dd_set_datetime(LogDriver *d, const gchar *field); gboolean afmongodb_dd_check_address(LogDriver *d, gboolean local); -- 1.7.10.4

2 4

Replace hostname with a hostname extracted from the message
by Clayton Dukes 14 Nov '12

14 Nov '12

Hi Folks, How can I extract a hostname from the message and replace the source ip with that name/ip address? Sample message: Nov 13 18:11:00 myhostname mymessage: Original Host=192.168.6.3:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions So, I need a rewrite rule(?) to take 192.168.6.3 and replace $HOST with it: Nov 13 18:11:00 92.168.6.3 mymessage:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 6

[RFC]: value-pairs key rewriting outside of key()
by Gergely Nagy 14 Nov '12

14 Nov '12

Hi! One thing I realized just now, is that there was a slight design mistake made when doing the value-pairs() key rewriting functionality: it was baked into the key() statement, so that as of this writing, one can only rewrite keys that were explicitly selected. While this sounds good on paper, it has one major drawback: If I want to rewrite keys that were selected by the scope, I have to explicitly select those keys myself. This makes a lot of things far more complicated than they need be, and this is why there's a hack in the mongodb destination that replaces leading dots with an underscore. Instead of doing that, I'd like to move the mongodb destination to use the key rewriting functionality, and while there, also teach $(format-json) to do something similar. Fortunately, it's only the config parser that needs an update, as the code below it was written in a way that allows me to do what I want. So, I'd like to add another way to configure key rewriting: value-pairs(scope(nv-pairs) rekey(".*", shift(1), add-prefix("_"))) This is very similar to the already existing way, except it's not inside a key(), but in the main value-pairs() body, thus, it will not add extra keys to the set, but will apply the transformations to keys already in it. For format-json, it would be fairly similar: $(format-json --scope nv-pairs --rekey .* --shift 1 --add-prefix "_") This functionality will then make it possible to rewrite keys that were implicitly added to the set by the scope, which in turn would make me much happier. As an FYI, I'll push this change to my merge-queue/3.4 branch tomorrow, unless anyone objects to it, or has a better suggestion by then. -- |8]

1 0

[Bug 205] New: Docs for macro MSGHDR incorrect
by bugzilla＠bugzilla.balabit.com 10 Nov '12

10 Nov '12

https://bugzilla.balabit.com/show_bug.cgi?id=205 Summary: Docs for macro MSGHDR incorrect Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 The docs state; MSGHDR Description: The name and the pid of the program that sent the log message in PROGRAM: PID format. Includes a trailing whitespace. Note that the macro returns an empty value if both the program and pid fields of the message are empty. When this macro is used the format is PROGRAM[PID]: including a trailing space. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[Bug 204] New: dbparser docs not clear for @NUMBER@
by bugzilla＠bugzilla.balabit.com 10 Nov '12

10 Nov '12

https://bugzilla.balabit.com/show_bug.cgi?id=204 Summary: dbparser docs not clear for @NUMBER@ Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: packaging AssignedTo: folti(a)balabit.hu ReportedBy: erempel(a)uvic.ca Type of the Report: --- Estimated Hours: 0.0 the @NUMBER@ parser should state that a leading minus sign is handled for non-hex numbers and that commas are not permitted. Alternatively, making @NUMBER@ accept commas would be nice, or allowing parameters for additional characters, although comma would be the only one I could think of. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5

Debugging tools for syslog-ng
by Anton Koldaev 10 Nov '12

10 Nov '12

Hi, I wanted to know if syslog-ng developers has some tools like mysqltuner or just a shell scripts to check syslog-ng configuration and get some recommendations on tuning? There is a lot of information about all the buffers/window sizes, etc. and it's very complicated to set the proper values if I have a bunch of different sources/destinations. For example if I'm using flow-control+multiple destinations it can stop reading the source at any time and I have no idea when and why it's happening and which value should I tune. -- Best regards, Koldaev Anton

3 12

rsyslog client produces "Error processing log message"
by Andreas Heinlein 08 Nov '12

08 Nov '12

Hello, we have a centralised log server running syslog-ng 3.1 OSE on Debian 6.0. On the client side, we were using syslog-ng but now I'd like to use rsyslog instead (for several reasons). Transport should be TLS-encrypted TCP. I have set up a connection between the two, but apparently syslog-ng fails to parse the log messages sent by rsyslog. Every log line goes like this: Nov 6 11:15:31 admin2-desktop syslog-ng[1578]: Error processing log message: <13>Nov 6 11:15:31 admin2-desktop ah: Test4 Does anyone have an idea what to configure with either rsyslog or syslog-ng so the two understand each other? Relevant server side config: source s_all { syslog(ip(172.16.x.x) port(6514) max_connections(50) tls( key_file("/etc/ssl/private/xxx.pem") cert_file("/etc/ssl/private/xxx.pem") ca_dir("/etc/ssl/certs") peer_verify(optional-untrusted) ) ); }; Relevant client side config: $DefaultNetstreamDriverCAFile /etc/ssl/certs/xxx.pem $DefaultNetstreamDriver gtls # use gtls netstream driver $ActionSendStreamDriverMode 1 # require TLS for the connection $ActionSendStreamDriverAuthMode anon # server is NOT authenticated *.* @@(o)mail:6514;RSYSLOG_TraditionalForwardFormat # (o) seems to be neccessary or syslog-ng will not even accept the connection; RSYSLOG_TraditionalForwardFormat apparently doesn't matter Thanks, Andreas

3 7

syslog-ng Insider - November 2012
by Peter Czanik 08 Nov '12

08 Nov '12

Dear syslog-ng users, This is the 18th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS syslog-ng 3.3.7 is released --------------------------- Version 3.3.7 was released last week, right before the “All Saints Day” holidays. It has many fixes for FreeBSD, so it should compile with the bundled ivykis on FreeBSD 8 and 9. Patches to compile it on FreeBSD 7 will also be included in http://www.freshports.org/sysutils/syslog-ng/ Packages for all supported or development versions of openSUSE are also available: https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 Packages for Debian and Ubuntu are available at http://asylum.madhouse-project.org/projects/debian/ Announcement for 3.3.7: https://lists.balabit.hu/pipermail/syslog-ng/2012-October/019602.html syslog-ng @ conferences ----------------------- syslog-ng was presented at two conferences in October. Fedora users and developers gathered in Paris: https://czanik.blogs.balabit.com/2012/10/fudcon-fedora-and-syslog-ng/ while openSUSE, SLES and Gentoo developers had their conference in Prague at the technical university: https://czanik.blogs.balabit.com/2012/10/osc-opensuse-and-syslog-ng/ syslog-ng 3.4 news ------------------ Many new features were added during the previous month. AMQP destination, syslog parser, a couple of new parsers for patterndb contributed by Evan Rempel and a few more smaller features and fixes. For a complete list check https://github.com/balabit/syslog-ng-3.4/commits/master NEW RELEASES: * syslog-ng OSE 3.3.7: https://lists.balabit.hu/pipermail/syslog-ng/2012-October/019602.html ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0