- syslog-ng - lists.balabit.hu

SYSLOG-NG 3.4 exiting with error code 2
by Deepak Gaur 27 Nov '12

27 Nov '12

Hello, I have cross-compiled syslog-ng for ARM target but it is exiting without showing any error code (on target). I am attaching strace output (syslog.txt). If anyone can help me out # syslog-ng -V syslog-ng 3.4.0alpha3 Installer-Version: 3.4.0alpha3 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.4#master#709ab1252920063e6d4d0d610ad5c1ee5ebdfb75 Compile-Date: Nov 27 2012 14:59:53 Available-Modules: syslogformat,afsocket-notls,afuser,basicfuncs,confgen,dbparser,tfuuid,afmongodb,afsocket-tls,afprog,afsocket,affile,csvparser Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-Linux-Caps: off Enable-Pcre: on STRACE OUTPUT ------------------------ strace syslog-ng execve("/sbin/syslog-ng", ["syslog-ng"], [/* 15 vars */]) = 0 brk(0) = 0x13000 uname({sys="Linux", node="192.168.119.78", ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001d000 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/v5l/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/v5l", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/tls/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/tls", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/v5l/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/v5l", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/fast-mult/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/fast-mult/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/fast-mult/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/fast-mult", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/half/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib/half", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/home/tbfsh/target//lib/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = -1 ENOENT (No such file or directory) stat64("/home/tbfsh/target//lib", 0x8ec40470) = -1 ENOENT (No such file or directory) open("/etc/ld.so.cache", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0644, st_size=10371, ...}) = 0 mmap2(NULL, 10371, PROT_READ, MAP_PRIVATE, 3, 0) = 0x4001e000 close(3) = 0 open("/lib/libsyslog-ng-3.4.0alpha3.so", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\3407\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1513316, ...}) = 0 mmap2(NULL, 496856, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40026000 mprotect(0x40093000, 28672, PROT_NONE) = 0 mmap2(0x4009a000, 20480, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6c) = 0x4009a000 mmap2(0x4009f000, 1240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4009f000 close(3) = 0 open("/lib/libnsl.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0004/\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=98605, ...}) = 0 mmap2(NULL, 116464, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400a0000 mprotect(0x400b2000, 28672, PROT_NONE) = 0 mmap2(0x400b9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11) = 0x400b9000 mmap2(0x400bb000, 5872, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x400bb000 close(3) = 0 open("/lib/libgmodule-2.0.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0@\r\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=18254, ...}) = 0 mmap2(NULL, 45268, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400bd000 mprotect(0x400c0000, 28672, PROT_NONE) = 0 mmap2(0x400c7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0x400c7000 close(3) = 0 open("/lib/libgthread-2.0.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0000\5\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=7368, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40021000 mmap2(NULL, 36912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400c9000 mprotect(0x400ca000, 28672, PROT_NONE) = 0 mmap2(0x400d1000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x400d1000 close(3) = 0 open("/lib/librt.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\300\26\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=39970, ...}) = 0 mmap2(NULL, 57872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400d3000 mprotect(0x400d9000, 28672, PROT_NONE) = 0 mmap2(0x400e0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x5) = 0x400e0000 close(3) = 0 open("/lib/libglib-2.0.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\270D\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1762078, ...}) = 0 mmap2(NULL, 1584268, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x400e2000 mprotect(0x4025c000, 28672, PROT_NONE) = 0 mmap2(0x40263000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x179) = 0x40263000 close(3) = 0 open("/lib/libevtlog.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\300\20\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=23326, ...}) = 0 mmap2(NULL, 49472, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40265000 mprotect(0x40269000, 28672, PROT_NONE) = 0 mmap2(0x40270000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3) = 0x40270000 close(3) = 0 open("/lib/libpcre.so.1", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\250\16\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=177343, ...}) = 0 mmap2(NULL, 200880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40272000 mprotect(0x4029b000, 28672, PROT_NONE) = 0 mmap2(0x402a2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28) = 0x402a2000 close(3) = 0 open("/lib/libdl.so.2", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\34\t\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=15728, ...}) = 0 mmap2(NULL, 41136, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x402a4000 mprotect(0x402a6000, 28672, PROT_NONE) = 0 mmap2(0x402ad000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0x402ad000 close(3) = 0 open("/lib/libpthread.so.0", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\330B\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=126565, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40022000 mmap2(NULL, 123376, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x402af000 mprotect(0x402c3000, 28672, PROT_NONE) = 0 mmap2(0x402ca000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x13) = 0x402ca000 mmap2(0x402cc000, 4592, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x402cc000 close(3) = 0 open("/lib/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\330Q\1\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0755, st_size=1498068, ...}) = 0 mmap2(NULL, 1217048, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x402ce000 mprotect(0x403eb000, 28672, PROT_NONE) = 0 mmap2(0x403f2000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x11c) = 0x403f2000 mmap2(0x403f5000, 8728, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x403f5000 close(3) = 0 open("/lib/libiconv.so.2", O_RDONLY) = 3 read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0(\0\1\0\0\0\230\22\0\0004\0\0\0"..., 512) = 512 fstat64(3, {st_mode=S_IFREG|0644, st_size=1061246, ...}) = 0 mmap2(NULL, 1048764, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x403f8000 mprotect(0x404f0000, 28672, PROT_NONE) = 0 mmap2(0x404f7000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xf7) = 0x404f7000 close(3) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40023000 mmap2(NULL, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x404f9000 set_tls(0x404f94a0, 0x40025674, 0x404fb5c8, 0x40025048, 0x40025048) = 0 mprotect(0x404f7000, 4096, PROT_READ) = 0 mprotect(0x403f2000, 8192, PROT_READ) = 0 mprotect(0x402ca000, 4096, PROT_READ) = 0 mprotect(0x402ad000, 4096, PROT_READ) = 0 mprotect(0x402a2000, 4096, PROT_READ) = 0 mprotect(0x40270000, 4096, PROT_READ) = 0 mprotect(0x40263000, 4096, PROT_READ) = 0 mprotect(0x400e0000, 4096, PROT_READ) = 0 mprotect(0x400d1000, 4096, PROT_READ) = 0 mprotect(0x400c7000, 4096, PROT_READ) = 0 mprotect(0x400b9000, 4096, PROT_READ) = 0 mprotect(0x4009a000, 8192, PROT_READ) = 0 mprotect(0x11000, 4096, PROT_READ) = 0 mprotect(0x40024000, 4096, PROT_READ) = 0 munmap(0x4001e000, 10371) = 0 set_tid_address(0x404f9048) = 577 set_robust_list(0x404f9050, 12) = 0 rt_sigaction(SIGRTMIN, {0x402b31e8, [], SA_SIGINFO|0x4000000}, NULL, 8) = 0 rt_sigaction(SIGRT_1, {0x402b2cb8, [], SA_RESTART|SA_SIGINFO|0x4000000}, NULL, 8) = 0 rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0 getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0 brk(0) = 0x13000 brk(0x34000) = 0x34000 gettimeofday({973374482, 838271}, NULL) = 0 open("/proc/stat", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001e000 read(3, "cpu 507 0 15055 364178 0 13 4 0"..., 1024) = 355 read(3, "", 1024) = 0 close(3) = 0 munmap(0x4001e000, 4096) = 0 open("/proc/stat", O_RDONLY) = 3 fstat64(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x4001e000 read(3, "cpu 507 0 15055 364205 0 13 4 0"..., 1024) = 355 read(3, "", 1024) = 0 close(3) = 0 munmap(0x4001e000, 4096) = 0 ioctl(0, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or TCGETS, {B115200 opost isig icanon echo ...}) = 0 ioctl(0, TIOCNOTTY) = 0 setsid() = 577 setrlimit(RLIMIT_NOFILE, {rlim_cur=4*1024, rlim_max=4*1024}) = 0 pipe([3, 4]) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x404f9048) = 578 close(4) = 0 read(3, "2\n", 6) = 2 close(3) = 0 exit_group(2) = ? +++ exited with 2 +++ -------------------------------------------------------------------------------- What could be the issue? Deepak Gaur

2 4

Queue size in syslog-ng?
by Abhijeet R 23 Nov '12

23 Nov '12

Hi all, In my setup, I send logs from all the servers to a central syslog server. This syslog-ng server also has a destination on a different port which is picked up by logstash's input block. (consider it just a consumer to those messages). What if this consumer goes down? How big is the queue? Is that configurable? How do I make sure that none of the messages are missed for the time the consumer is down? -- Cheers, Abhijeet R http://blog.abhijeetr.com

7 7

merging stuff from merge-queue/3.4
by Balazs Scheidler 22 Nov '12

22 Nov '12

Hi, I have started merging the branch $subject, however I've found some patches I couldn't merge. system-source: Use follow-freq(1) on FreeBSD prior to 9.1 Are you sure time based polling would work here? I mean time based polling also checks if the size of the file has changed, as it was meant to be used only with regular files. I'm afraid that the inability to use kqueue on /dev/klog would prevent using kqueue completely, as the syslog-ng file source code requires either regular files or pollable devices. dbparser: support inheriting properties in a corellation action I merged this one, although I have some doubts. It uses log_msg_clone_cow(), which does have some tricky semantics, as it assumes that the cloned message will not change, once cloned. I was thinking about removing clone_cow() completely because of this tricky stuff, and whenever we need a clone we're about to change the LogMessage anyway, and delaying the actual change means that we have to do two malloc() calls, instead of one. Can you test whether changing the message that triggered the action with a rewrite rule, keeps the cloned one intact? This would need a parallel path to the db-parser(). dbparser: Make the context length available to action conditions I didn't pick this one. I was thinking about using a template function to return the number of entries in the current context. This patch changes the log message while it could keep it read-only, and even if I wanted to do this macro-like, I'd have created a real macro. Both template formatting and template functions receive the context as arguments anyway, so the required information is readily available. -- Bazsi

5 8

[PATCH 0/3] Type hinting
by Gergely Nagy 22 Nov '12

22 Nov '12

The patches that will follow this overview implement type-hinting support into LogTemplate and value-pairs(), then extend the mongodb destination and format-json to use these hints. In the end, we'll do things like: mongodb(value-pairs(scope("selected-macros", "nv-pairs") pair("DATE", datetime("$UNIXTIME")) pair("PID", int("$PID")) pair(".bool", boolean("true")))); And this will use the appropriate types when inserting into mongodb. The $(format-json) template function works similarly: template("$(format-json --scope selected-macros,nv-pairs DATE=datetime($UNIXTIME) PID=int($PID) .bool=boolean(TRUE))\n"); I tried to make the code as performant as possible, there are no extra allocations performed, and in case there is no casting to perform, it's just an extra integer comparsion. It can be tweaked further, if so need be, but for now, I'm happy with the results. Nevertheless, there are a couple of things I'm not entirely satisfied with (namely, abusing the ScratchBuffers to store the type hint on the first byte; or the way the value-pairs CLI interface parses type hints), and I will fix them in the coming days. While using this isn't the most convenient thing ever, and setting default types is not possible from within drivers (technically, it is, but it's awkward), it's a step in the right direction. Once nvpairs gains proper type support, that'll make things a whole lot sweeter! This - and any future updates - are available from my feature/3.4/templates/cast-typehints branch.

2 9

RFC: syslog-parser
by Balazs Scheidler 22 Nov '12

22 Nov '12

Hi, I've figured it is not nice from me that I keep pushing stuff to "master" without getting proper review from the list. So here's a attempt to change that. I've just pushed an experimental patch to the "syslog-parser" branch on github that allows the syslog parsing functionality to be used as a syslog-ng parser: https://github.com/balabit/syslog-ng-3.4/tree/syslog-parser Here's a short description on why this is useful (quoting the commit message): This patch creates a new parser, to explicitly parse messages as syslog messages. This can be used to selectively parse these messages based on some filters. Imagine this use-case: log { source { udp(port(2000) flags(no-parse)); }; parser { log { junction { log { filter { netmask("127.0.0.1/32"); }; parser { syslog-parser(); }; flags(final); }; log { filter { netmask("127.0.0.2/32"); }; parser { csv-parser(columns("C1", "C2", "C3")); }; flags(final); }; }; }; }; destination { file("/home/bazsi/logs/qqq" template("$(format-json --key *)\n")); }; }; Messages from 127.0.0.1 will get parsed as syslog messages, while 127.0.0.2 as a csv-parser() style messages. I'd welcome any kind of feedback, code or functionality wise. I still have some stuff to fix in this, but after initial feedback I'm going to merge it to master. Thanks in advance. -- Bazsi

6 8

Filter netmask not working as expected
by Agus 20 Nov '12

20 Nov '12

Hi guys, Just new, and created the following conf for testing purposes. The problem is that i get the logs in both destinations despite the filter. @version: 3.4 @include "scl.conf" options { keep_hostname(yes); normalize_hostnames(yes); threaded(yes); ts_format(iso); # Adds TZ #use_fqdn(yes); use_dns(no); }; source s_local { system(); internal(); }; source s_network { udp(); }; destination d_local { file("/var/log/messages"); }; destination d_my_mac { file("/var/log/mymac"); }; filter f_my_mac { netmask(10.24.18.2/255.255.255.255); }; log { source(s_network); filter(f_my_mac); destination(d_my_mac); flags(final); }; log { source(s_local); # uncomment this line to open port 514 to receive messages source(s_network); #destination(d_central_udp); destination(d_local); }; as netmask i also tried cidr /24 and same thing. The problem is that i get the logs in both destinations. I only want to have them in my_mac Thanks!

2 2

[PATCH 1/2] [dbparser] inherit nvpairs from parent in correlation action
by balint.kovacs＠balabit.com 16 Nov '12

16 Nov '12

From: Balint Kovacs <blint(a)blint.hu> This patch implements an inherit-name-value-pairs attribute for message type actions is patterndb correlation. In case this is set to TRUE, the triggering log message is cloned and all name-value pairs are automagically copied to the emitted message. The values set in the values XML subtree overwrite the original values. In case it is set to FALSE or unspecified, a new message is generated and only the values set manually are added to it. <actions> <action> <message inherit-name-value-pairs='TRUE'> <values> <value name="PROGRAM">my-program-name</value> <value name="MSG">Hello new message!</value> <value name="contextid">${session-id}@2</value> </values> </message> </action> </actions> Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/patterndb-int.h | 1 + modules/dbparser/patterndb.c | 32 +++++++++++++++++++++++++++++--- 2 files changed, 30 insertions(+), 3 deletions(-) diff --git a/modules/dbparser/patterndb-int.h b/modules/dbparser/patterndb-int.h index 0434847..154b686 100644 --- a/modules/dbparser/patterndb-int.h +++ b/modules/dbparser/patterndb-int.h @@ -126,6 +126,7 @@ typedef struct _PDBAction guint8 content_type; guint16 rate; guint32 id:8, rate_quantum:24; + gboolean inherit_nvpairs; union { PDBMessage message; diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 8720ea0..168a24c 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -388,6 +388,17 @@ pdb_action_set_trigger(PDBAction *self, const gchar *trigger, GError **error) g_set_error(error, 0, 1, "Unknown trigger type: %s", trigger); } +void +pdb_action_set_inheritance(PDBAction *self, const gchar *inherit_nvpairs, GError **error) +{ + if (strcmp(inherit_nvpairs, "TRUE") == 0) + self->inherit_nvpairs = TRUE; + else if (strcmp(inherit_nvpairs, "FALSE") == 0) + self->inherit_nvpairs = FALSE; + else + g_set_error(error, 0, 1, "Unknown inheritance type: %s", inherit_nvpairs); +} + PDBAction * pdb_action_new(gint id) { @@ -398,6 +409,7 @@ pdb_action_new(gint id) self->trigger = RAT_MATCH; self->content_type = RAC_NONE; self->id = id; + self->inherit_nvpairs = FALSE; return self; } @@ -552,9 +564,18 @@ pdb_rule_run_actions(PDBRule *self, gint trigger, PatternDB *db, PDBContext *con case RAC_NONE: break; case RAC_MESSAGE: - genmsg = log_msg_new_empty(); - genmsg->flags |= LF_LOCAL; - genmsg->timestamps[LM_TS_STAMP] = msg->timestamps[LM_TS_STAMP]; + if (action->inherit_nvpairs) + { + LogPathOptions path_options = LOG_PATH_OPTIONS_INIT; + path_options.ack_needed = FALSE; + genmsg = log_msg_clone_cow(msg, &path_options); + } + else + { + genmsg = log_msg_new_empty(); + genmsg->flags |= LF_LOCAL; + genmsg->timestamps[LM_TS_STAMP] = msg->timestamps[LM_TS_STAMP]; + } if (context) { switch (context->key.scope) @@ -942,6 +963,11 @@ pdb_loader_start_element(GMarkupParseContext *context, const gchar *element_name } else if (strcmp(element_name, "message") == 0) { + for (i = 0; attribute_names[i]; i++) + { + if (strcmp(attribute_names[i], "inherit-name-value-pairs") == 0) + pdb_action_set_inheritance(state->current_action, attribute_values[i], error); + } if (!state->in_action) { *error = g_error_new(1, 0, "Unexpected <message> element, it must be inside an action"); -- 1.7.9.5

5 9

[PATCH (3.4)] afmongodb: Add support for datetime fields
by Gergely Nagy 15 Nov '12

15 Nov '12

For TTL collections primarly (but useful for other scenarios aswell), introduce a new option: datetime(FIELD_NAME). This allows one to mark a single field anywhere in the structure as a datetime field, and the driver will attempt to convert it to a proper datetime, by assuming it is a unix timestamp. If the conversion fails, or the field is not found, it will not be inserted. Usage: mongodb(datetime("some.date") value-pairs( key("MESSAGE") pair("some.date", "$UNIXTIME"))) This will result in something like: { "_id" : ObjectId("509b8ffc1a1006213a000001"), "some" : { "date" : ISODate("2012-11-08T10:56:59Z") }, "MESSAGE" : "foobar" } Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb-grammar.ym | 2 ++ modules/afmongodb/afmongodb-parser.c | 1 + modules/afmongodb/afmongodb.c | 61 +++++++++++++++++++++++++++++--- modules/afmongodb/afmongodb.h | 1 + 4 files changed, 61 insertions(+), 4 deletions(-) diff --git a/modules/afmongodb/afmongodb-grammar.ym b/modules/afmongodb/afmongodb-grammar.ym index f108f26..40eccc0 100644 --- a/modules/afmongodb/afmongodb-grammar.ym +++ b/modules/afmongodb/afmongodb-grammar.ym @@ -53,6 +53,7 @@ extern ValuePairsTransformSet *last_vp_transset; %token KW_SERVERS %token KW_SAFE_MODE %token KW_PATH +%token KW_DATETIME %% @@ -97,6 +98,7 @@ afmongodb_option | KW_USERNAME '(' string ')' { afmongodb_dd_set_user(last_driver, $3); free($3); } | KW_PASSWORD '(' string ')' { afmongodb_dd_set_password(last_driver, $3); free($3); } | KW_SAFE_MODE '(' yesno ')' { afmongodb_dd_set_safe_mode(last_driver, $3); } + | KW_DATETIME '(' string ')' { afmongodb_dd_set_datetime(last_driver, $3); free($3); } | value_pair_option { afmongodb_dd_set_value_pairs(last_driver, $1); } | dest_driver_option ; diff --git a/modules/afmongodb/afmongodb-parser.c b/modules/afmongodb/afmongodb-parser.c index 8aa55c1..1c51154 100644 --- a/modules/afmongodb/afmongodb-parser.c +++ b/modules/afmongodb/afmongodb-parser.c @@ -39,6 +39,7 @@ static CfgLexerKeyword afmongodb_keywords[] = { { "host", KW_HOST }, { "port", KW_PORT }, { "path", KW_PATH }, + { "datetime", KW_DATETIME }, { NULL } }; diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 205ab98..bb267a5 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -22,6 +22,7 @@ */ #include <time.h> +#include <stdlib.h> #include "afmongodb.h" #include "afmongodb-parser.h" @@ -32,6 +33,7 @@ #include "nvtable.h" #include "logqueue.h" #include "value-pairs.h" +#include "scratch-buffers.h" #include "mongo.h" @@ -55,6 +57,7 @@ typedef struct gint port; gboolean safe_mode; + gchar *datetime_field; gchar *user; gchar *password; @@ -207,6 +210,15 @@ afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state) self->safe_mode = state; } +void +afmongodb_dd_set_datetime(LogDriver *d, const gchar *field) +{ + MongoDBDestDriver *self = (MongoDBDestDriver *)d; + + g_free(self->datetime_field); + self->datetime_field = g_strdup(field); +} + /* * Utilities */ @@ -343,7 +355,7 @@ afmongodb_vp_obj_end(const gchar *name, if (prev_data) root = (bson *)*prev_data; else - root = (bson *)user_data; + root = (bson *)(((gpointer *)user_data)[0]); if (prefix_data) { @@ -357,6 +369,30 @@ afmongodb_vp_obj_end(const gchar *name, } static gboolean +afmongodb_field_is_datetime(const gchar *prefix, const gchar *name, + const gchar *datetime_field) +{ + size_t plen = 0; + + if (!datetime_field) + return FALSE; + + if (prefix) + plen = strlen(prefix); + + if (strncmp(datetime_field, prefix, plen) != 0) + return FALSE; + + if (datetime_field[plen] != '.') + return FALSE; + + if (strcmp(&datetime_field[plen + 1], name) != 0) + return FALSE; + + return TRUE; +} + +static gboolean afmongodb_vp_process_value(const gchar *name, const gchar *prefix, const gchar *value, gpointer *prefix_data, gpointer user_data) @@ -366,7 +402,7 @@ afmongodb_vp_process_value(const gchar *name, const gchar *prefix, if (prefix_data) o = (bson *)*prefix_data; else - o = (bson *)user_data; + o = (bson *)(((gpointer *)user_data)[0]); if (name[0] == '.') { @@ -378,7 +414,20 @@ afmongodb_vp_process_value(const gchar *name, const gchar *prefix, bson_append_string (o, tx_name, value, -1); } else - bson_append_string (o, name, value, -1); + { + MongoDBDestDriver *self = (MongoDBDestDriver *)(((gpointer *)user_data)[1]); + + if (afmongodb_field_is_datetime(prefix, name, self->datetime_field)) + { + gchar *endptr; + gint64 datetime = (gint64)strtoul(value, &endptr, 10) * 1000; + + if (endptr[0] == '\0') + bson_append_utc_datetime(o, name, datetime); + } + else + bson_append_string (o, name, value, -1); + } return FALSE; } @@ -390,6 +439,7 @@ afmongodb_worker_insert (MongoDBDestDriver *self) guint8 *oid; LogMessage *msg; LogPathOptions path_options = LOG_PATH_OPTIONS_INIT; + gpointer user_data[] = { NULL, self }; afmongodb_dd_connect(self, TRUE); @@ -408,11 +458,13 @@ afmongodb_worker_insert (MongoDBDestDriver *self) bson_append_oid (self->bson, "_id", oid); g_free (oid); + user_data[0] = self->bson; + value_pairs_walk(self->vp, afmongodb_vp_obj_start, afmongodb_vp_process_value, afmongodb_vp_obj_end, - msg, self->seq_num, self->bson); + msg, self->seq_num, user_data); bson_finish (self->bson); if (!mongo_sync_cmd_insert_n(self->conn, self->ns, 1, @@ -650,6 +702,7 @@ afmongodb_dd_free(LogPipe *d) string_list_free(self->servers); if (self->vp) value_pairs_free(self->vp); + g_free(self->datetime_field); log_dest_driver_free(d); } diff --git a/modules/afmongodb/afmongodb.h b/modules/afmongodb/afmongodb.h index 6a61616..76f8b83 100644 --- a/modules/afmongodb/afmongodb.h +++ b/modules/afmongodb/afmongodb.h @@ -39,6 +39,7 @@ void afmongodb_dd_set_password(LogDriver *d, const gchar *password); void afmongodb_dd_set_value_pairs(LogDriver *d, ValuePairs *vp); void afmongodb_dd_set_safe_mode(LogDriver *d, gboolean state); void afmongodb_dd_set_path(LogDriver *d, const gchar *path); +void afmongodb_dd_set_datetime(LogDriver *d, const gchar *field); gboolean afmongodb_dd_check_address(LogDriver *d, gboolean local); -- 1.7.10.4

2 4

Replace hostname with a hostname extracted from the message
by Clayton Dukes 14 Nov '12

14 Nov '12

Hi Folks, How can I extract a hostname from the message and replace the source ip with that name/ip address? Sample message: Nov 13 18:11:00 myhostname mymessage: Original Host=192.168.6.3:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions So, I need a rewrite rule(?) to take 192.168.6.3 and replace $HOST with it: Nov 13 18:11:00 92.168.6.3 mymessage:LOW_THRESHOLD_EVENT - 0 AUTHORIZED sessions ______________________________________________________________ Clayton Dukes ______________________________________________________________

2 6

[RFC]: value-pairs key rewriting outside of key()
by Gergely Nagy 14 Nov '12

14 Nov '12

Hi! One thing I realized just now, is that there was a slight design mistake made when doing the value-pairs() key rewriting functionality: it was baked into the key() statement, so that as of this writing, one can only rewrite keys that were explicitly selected. While this sounds good on paper, it has one major drawback: If I want to rewrite keys that were selected by the scope, I have to explicitly select those keys myself. This makes a lot of things far more complicated than they need be, and this is why there's a hack in the mongodb destination that replaces leading dots with an underscore. Instead of doing that, I'd like to move the mongodb destination to use the key rewriting functionality, and while there, also teach $(format-json) to do something similar. Fortunately, it's only the config parser that needs an update, as the code below it was written in a way that allows me to do what I want. So, I'd like to add another way to configure key rewriting: value-pairs(scope(nv-pairs) rekey(".*", shift(1), add-prefix("_"))) This is very similar to the already existing way, except it's not inside a key(), but in the main value-pairs() body, thus, it will not add extra keys to the set, but will apply the transformations to keys already in it. For format-json, it would be fairly similar: $(format-json --scope nv-pairs --rekey .* --shift 1 --add-prefix "_") This functionality will then make it possible to rewrite keys that were implicitly added to the set by the scope, which in turn would make me much happier. As an FYI, I'll push this change to my merge-queue/3.4 branch tomorrow, unless anyone objects to it, or has a better suggestion by then. -- |8]

1 0