September 2012 - syslog-ng - lists.balabit.hu

[Bug 194] New: Can't open libsyslog-ng-crypto. so during module loading on Fedora
by bugzilla＠bugzilla.balabit.com 27 Oct '12

27 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=194 Summary: Can't open libsyslog-ng-crypto.so during module loading on Fedora Product: syslog-ng Version: 3.3.x Platform: All OS/Version: Linux Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu CC: algernon(a)balabit.hu Type of the Report: --- Estimated Hours: 0.0 This clone is to track the library path issue separately. +++ This bug was initially created as a clone of Bug #188 +++ Linux distro: Fedora 17 x86_64 Syslog-ng source tarball: syslog-ng-3.3.5.90-20120810-v3.3.5-43-g19f6c.tar.gz we have encountered a couple of problems when running syslog-ng-3.3.5.90-20120810 under Fedora 17 x86_64: 1) ld.so library path 2) undefined symbol: tls_context_setup_session Steps to reproduce the problem: 1) just run "/usr/sbin/syslog-ng --version" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Error opening plugin module; module='afsocket-tls', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' Available-Modules: dummy,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- The library path problem can be solved by creating the file "/etc/ld.so.conf.d/syslog-ng.conf" containing the syslog-ng module directory ( /usr/lib64/syslog-ng in this particular case ). Running syslog-ng under strace without touching the ld.so library paths produces: ---------- # # strace -e trace=file /usr/sbin/syslog-ng --version ... access("//usr/lib64/syslog-ng/libafsocket.so", F_OK) = 0 stat("//usr/lib64/syslog-ng/libafsocket.so", {st_mode=S_IFREG|0755, st_size=103560, ...}) = 0 open("//usr/lib64/syslog-ng/libafsocket.so", O_RDONLY|O_CLOEXEC) = 4 open("/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/tls/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/tls", {st_mode=S_IFDIR|0555, st_size=4096, ...}) = 0 open("/usr/lib64/x86_64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64/x86_64", 0x7fff0f81fcb0) = -1 ENOENT (No such file or directory) open("/usr/lib64/libsyslog-ng-crypto.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory) stat("/usr/lib64", {st_mode=S_IFDIR|0555, st_size=49152, ...}) = 0 Error opening plugin module; module='afsocket', error='libsyslog-ng-crypto.so: cannot open shared object file: No such file or directory' ... ----------- Note that all syslog-ng modules with the exception of libsyslog-ng-crypto.so are loaded from the syslog-ng modules directory. Output of "syslog-ng --version" after creating the file "/etc/ld.so.conf.d/syslog-ng.conf" ---------- # /usr/sbin/syslog-ng --version syslog-ng 3.3.5.90 Installer-Version: 3.3.5.90 Revision: Compile-Date: Aug 17 2012 22:09:52 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afsql Error opening plugin module; module='afsocket-notls', error='//usr/lib64/syslog-ng/libafsocket-notls.so: undefined symbol: tls_context_setup_session' Available-Modules: dummy,afsocket,afsql,tfjson,dbparser,csvparser,convertfuncs,afuser,basicfuncs,afmongodb,afsocket-tls,confgen,syslogformat,affile,afprog Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on ---------- /jpo -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

GeoIP template function
by Csaba Major 19 Oct '12

19 Oct '12

Hi, I have a system where I have a lot of logs with IP addresses (typically firewall logs), and the requirement was to also include the country name in the logs. I found that a geoip based template function would be an easy way for this, so I created a PoC for that. It works pretty well, however some improvements and performance measurement would be necessary. The created patch is attached (based on 3.3.5), some may find it useful :) (It requires libgeoip, and a geoip database, of course) Regards, Csaba

4 6

[Bug 151] New: syslog-ng opens /dev/ log device on every HUP without closing it first
by bugzilla＠bugzilla.balabit.com 08 Oct '12

08 Oct '12

https://bugzilla.balabit.com/show_bug.cgi?id=151 Summary: syslog-ng opens /dev/log device on every HUP without closing it first Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Solaris Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: michael.hocke(a)nyu.edu Type of the Report: bug Estimated Hours: 0.0 On Solaris 10 the /dev/log device can be held open concurrently for up to only 16 instances. syslog-ng opens a new instance of /dev/log after HUPing the process, i.e. after the 16th HUP syslog-ng quits because the new configuration fails and so does the previous configuration. This has been working fine in 3.0. I cannot test if this started to happen in 3.1, 3.2, or if this is new for 3.3. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 8

Syslog stops writing logs with extra [SQL] destination
by Anton Koldaev 05 Oct '12

05 Oct '12

Hello, I have the same "expr" logs going to file AND sql. With file-only logging it works great. When I'm enabling SQL logging it stops writing both to file and to sql ~1-10min(from 10 000 to 200 000 lines) after start. Every time there are no dropped messages, but 1000 stored for SQL location. syslog-ng 3.3.6 - number of nodes: ~150 - max-connections(200) - log_iw_size(20000) - log_fetch_limit(100) - log_fifo_size (400000); - flush_lines(100) - flush_timeout(100) - flags(flow-control) # the same issue with and without flow control syslog-server ~ % sudo syslog-ng-ctl stats | sort -rnk6 -t ";" | grep expr dst.file;d_app_expr#0;/logs/app/expr.log;o;processed;30457 destination;d_app_expr;;a;processed;30457 destination;d_mysql_expr;;a;processed;30003 * dst.sql;d_mysql_expr#0;mysql,mysql-db-ip,port,syslog_production,expr_${HOST}_${R_Y;a;stored;1000 * dst.sql;d_mysql_expr#0;mysql,mysql-db-ip,port,syslog_production,expr_${HOST}_${R_Y;a;dropped;0 dst.file;d_app_expr#0;/logs/app/expr.log;o;stored;0 dst.file;d_app_expr#0;/logs/app/expr.log;o;dropped;0 I see no errors in syslog-ng debug logs or mysql error log. After restarting syslog-ng it writes both logs for some time and stops again. I will appreciate any help. Additional question: is there any mysqltuner.pl-like tool for syslog-ng? I don't feel happy to tune all those buffers and windows manually with no being sure it will help. -- Best regards, Koldaev Anton

1 1

how to convert multiline message to single line?
by Sergey V. Lobanov 03 Oct '12

03 Oct '12

Hello, I'd like to see multiline syslog messages as one line. Is it possible via syslog-ng 3.0? -- wbr, Sergey V. Lobanov Timezone: MSK(GMT+4)

4 6

[Bug 199] New: syslog-ng 3.3.6. 90 sometimes exits on SIGHUP on Solaris 10 SPARC
by bugzilla＠bugzilla.balabit.com 28 Sep '12

28 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=199 Summary: syslog-ng 3.3.6.90 sometimes exits on SIGHUP on Solaris 10 SPARC Product: syslog-ng Version: 3.3.x Platform: Other OS/Version: Solaris Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: irwin(a)princeton.edu Type of the Report: bug Estimated Hours: 0.0 Created an attachment (id=66) --> (https://bugzilla.balabit.com/attachment.cgi?id=66) configure.out Version: syslog-ng 3.3.6.90 Platform: Solaris 10 on SPARC I've been running syslog-ng 3.0.6 fine for several years. I've been trying to upgrade to 3.3.latest. When I upgraded to 3.3.6, syslog-ng would sometimes quit and dump core unexpectedly when I sent it SIGHUP as part of log file rotation. I figured it might be an ivykis issue perhaps related to https://bugzilla.balabit.com/show_bug.cgi?id=190 , so I've just tried upgrading to 3.3.6.90. But the error I see remains the same. Log message: Sep 26 03:00:03 example.org supervise/syslog-ng[23280]: Daemon exited due to a deadlock/signal/failure, restarting; exitcode='131' Here's the core dump's backtrace: gdb /usr/local/sbin/syslog-ng /var/core/core.syslog-ng.0.23281 GNU gdb (GDB) 7.4 ... Reading symbols from /var/local/sbin/syslog-ng...done. [New LWP 1] [New LWP 2] [New LWP 3] [Thread debugging using libthread_db enabled] [New Thread 1 (LWP 1)] [New Thread 2 (LWP 2)] [New Thread 3 (LWP 3)] Core was generated by `/usr/local/sbin/syslog-ng --cfgfile /etc/syslog-ng/main.conf --pidfile /var/run'. Program terminated with signal 3, Quit. #0 0xfef4e068 in kill () from /lib/libc.so.1 (gdb) bt #0 0xfef4e068 in kill () from /lib/libc.so.1 #1 0xff2b4c24 in main_loop_reload_config_apply () at mainloop.c:518 #2 0xff2b4640 in main_loop_io_worker_sync_call (func=0xff2b4a20 <main_loop_reload_config_apply>) at mainloop.c:455 #3 0xff2d68c8 in iv_signal_event (_this=0xff30ddfc) at iv_signal.c:169 #4 0xff2d62b8 in iv_event_raw_got_event (_this=0xff30de20) at iv_event_raw.c:87 #5 0xff2d3d38 in iv_run_active_list (active=<optimized out>, st=<optimized out>) at iv_main.c:235 #6 iv_main () at iv_main.c:279 #7 0xff2b4474 in main_loop_run () at mainloop.c:735 #8 0x00011538 in main (argc=1, argv=0xffbffd4c) at main.c:263 -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 6

Problem with csv-parser
by Tracy Dalzell 26 Sep '12

26 Sep '12

I am having a problem with the csv-parser. I expect the parser to not include the single quotes in the MACRO (column). It does not include the single quotes for the first column (id) but they are included for every column after the first one. Relevent details below: parser p_apistats { db-parser(file("/etc/syslog-ng/apilogs.xml"));}; parser p_apistats returns ${VALUES} = 'b8144eea3646a7bc47169e0b9cc5f575', '555.55.55.555', 'web07.myhost.com' parser p_apistatscsv { csv-parser(columns(id, ip, host) delimiters(",") flags(greedy,strip-whitespace) template("${VALUES}")); }; destination d_test { file("/var/log/splunk/$HOST" template("{ \"id\" : \"${id}\" \"ip\" : \"${ip}\" \"host\" : \"${host}\" }\n")); }; log { source(s_remote); parser(p_apistats); parser(p_apistatscsv); destination(d_test); }; Results: { "id" : "b8144eea3646a7bc47169e0b9cc5f575" "ip" : "'555.55.55.555'" "host" : "'web07.myhost.com'" }

1 0

[Bug 197] New: Characters lost when sending message over network that begins with {
by bugzilla＠bugzilla.balabit.com 25 Sep '12

25 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=197 Summary: Characters lost when sending message over network that begins with { Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: bfield.wrk(a)gmail.com Type of the Report: --- Estimated Hours: 0.0 When sending a message that begins with { over the network the leading parenthesis and white space are lost. I've tested this on 3.3.4 with a manually compiled version and the version distributed with Ubuntu. Both are impacted by this bug. Here is a simple configuration to try: --------- syslog-ng.conf --------- source s_src { system(); }; source s_udp { udp(ip(127.0.0.1)); }; destination d_testfile { file("/var/log/test_json" template("$MSG\n")); }; destination d_udp_addparen { udp("127.0.0.1" template("{ second: { $MSG\n")); }; destination d_filedirect { file("/var/log/test_jsondirect" template("{ second: { $MSG\n")); }; log { source(s_src); destination(d_udp_addparen); destination(d_filedirect); }; log { source(s_udp); destination(d_testfile); }; -------- end syslog-ng.conf ------- # echo "{ json type message }" | logger # tail /var/log/test_jsondirect { second: { { json type message } # tail /var/log/test_json second: { { json type message } -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 2

[PATCH] system-source: Correct a typo in the unix-dgram() part
by Gergely Nagy 25 Sep '12

25 Sep '12

From: Gergely Nagy <algernon(a)madhouse-project.org> The option is called perm(), not perms(). Reported-by: Peter Czanik <czanik(a)balabit.hu> Signed-off-by: Gergely Nagy <algernon(a)madhouse-project.org> --- modules/system-source/system-source.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/system-source/system-source.c b/modules/system-source/system-source.c index 9ba0480..929a5f8 100644 --- a/modules/system-source/system-source.c +++ b/modules/system-source/system-source.c @@ -41,7 +41,7 @@ system_sysblock_add_unix_dgram(GString *sysblock, const gchar *path, { g_string_append_printf(sysblock, "unix-dgram(\"%s\"", path); if (perms) - g_string_append_printf(sysblock, " perms(%s)", perms); + g_string_append_printf(sysblock, " perm(%s)", perms); g_string_append(sysblock, ");\n"); } -- 1.7.10

1 0

[Bug 196] New: syslog-ng fails to compile with json-c >= 0.10
by bugzilla＠bugzilla.balabit.com 25 Sep '12

25 Sep '12

https://bugzilla.balabit.com/show_bug.cgi?id=196 Summary: syslog-ng fails to compile with json-c >= 0.10 Product: syslog-ng Version: 3.3.x Platform: All OS/Version: All Status: ASSIGNED Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: algernon(a)balabit.hu ReportedBy: algernon(a)balabit.hu Type of the Report: bug Estimated Hours: 0.0 The json-c 0.10 release changed how it serializes data, and that in turn broke syslog-ng, as we relied on the old not-all-that-public interface. Bumping the version requirement to >= 0.10 would not be wise, so syslog-ng should support both. I have a fix already, but that currently depends on other work I did starting from the 3.4 branch, so it needs to be backported to 3.3. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 5