December 2011 - syslog-ng - lists.balabit.hu

[Bug 142] New: --worker-threads=0 support
by bugzilla＠bugzilla.balabit.com 25 Jan '12

25 Jan '12

https://bugzilla.balabit.com/show_bug.cgi?id=142 Summary: --worker-threads=0 support Product: syslog-ng Version: 3.3.x Platform: PC OS/Version: Linux Status: NEW Severity: normal Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: arekm(a)maven.pl Type of the Report: enhancement Estimated Hours: 0.0 Hello, I was hoping that threading support would solve my problem described in https://bugzilla.balabit.com/show_bug.cgi?id=113. Unfortunately it looks like it is impossible to create configuration for which each target will use one thread by default (as only such configuration is safe in case of some targets blocking on I/O). This feature request is about adding some option like: --worker-threads=0 that would limit number of threads to numer of sources + number of destinations, so each possible blocking I/O would be happening in separate thread. Yes, it would be more than cpu cores but this configuration is about reliability and not performance. (right now I have only one, network logging target that needs to always work, so probably disabling threading globally and enabling threading only for this network target would get me "always working" logging via net, even if some infinite I/O blocking happens. What would be best is to have ability to have such config on by default for all targets) -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 12

Hello, and project for Win32 port
by Ramón García Fernández 16 Jan '12

16 Jan '12

Hello, I am Ramón García Fernández and I work for the IT department of the Comisión Nacional de Mercado de Valores, the Spanish regulator of securities. Would you be interested in an opensource Win32 port of syslog-ng? Best regards, Aviso legal – Comisión Nacional del Mercado de Valores Este mensaje y, en su caso, los ficheros que lleve incorporados, está dirigido exclusivamente a su destinatario y es de carácter confidencial. Si fuere recibido por error o se tuviere conocimiento del mismo sin ser su destinatario, rogamos nos lo comunique por la misma vía o telefónicamente (91 585 15 00) y proceda a su destrucción, debiendo abstenerse de utilizar, transmitir, divulgar o reproducir la información contenida en el mismo. La CNMV se reserva las acciones legales que procedan contra todo tercero que acceda de forma ilegítima al contenido de cualquier mensaje externo procedente de la entidad Para información y consultas visite nuestra web: http://www.cnmv.es Disclaimer - Comisión Nacional del Mercado de Valores This message, its content and any file attached thereto is for the intended recipient only and is confidential. If you have received this e-mail in error or had access to it, you should note that the information in it is private and any use thereof is unauthorised. In such an event please notify us by e-mail or by telephone (+ 34 91 585 15 00). Any reproduction of this e-mail by whatsoever means and any transmission or dissemination thereof to other persons is prohibited. The Comisión Nacional del Mercado de Valores reserves the right to take legal action against any persons unlawfully gaining access to the content of any external message it has emitted For additional information, please visit our website: http://www.cnmv.es

3 4

Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu 10.10 (Luis Pugoy)
by Luis Pugoy 16 Jan '12

16 Jan '12

> > > Message: 1 > Date: Mon, 5 Dec 2011 19:54:42 +0800 > From: Luis Pugoy <lpugoy(a)insynchq.com> > Subject: Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu > 10.10 > To: syslog-ng(a)lists.balabit.hu > Message-ID: > <CA+WGxEhKYMQXbYTVUZVScjaXDdjRETGSCk5nZaisW9p-UOQWoA(a)mail.gmail.com > > > Content-Type: text/plain; charset="iso-8859-1" > > > > > Message: 3 > > Date: Mon, 05 Dec 2011 10:10:40 +0100 > > From: Gergely Nagy <algernon(a)balabit.hu> > > Subject: Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu > > 10.10 > > To: Syslog-ng users' and developers' mailing list > > <syslog-ng(a)lists.balabit.hu> > > Message-ID: <87aa77l6y7.fsf(a)algernon.balabit> > > Content-Type: text/plain > > > > Luis Pugoy <lpugoy(a)insynchq.com> writes: > > > > > Does anyone have any insight into this? The segfault is happening more > > > often now. :( > > > > I'd suggest either upgrading to 3.3 from my repo[1], or if an upgrade is > > not feasible, then if you could compile 3.1 with debug symbols, get it > > to drop a core, and do a backtrace, that would help tremendously. > > > > [1]: http://asylum.madhouse-project.org/projects/debian/ > > > > To do the latter, the following steps should work: > > > > $ apt-get source syslog-ng > > $ sudo apt-get build-dep syslog-ng > > $ cd syslog-ng-* > > $ DEB_BUILD_OPTIONS="debug nostrip" dpkg-buildpackage -us -uc -rfakeroot > > $ sudo dpkg -i ../syslog-ng_*.deb > > > > Then add --enable-core to SYSLOGNG_OPTS in /etc/default/syslog-ng, and > > when it segfaults, it'll make a core in (as far as I remember) /. Once > > that happened, do something like this: > > > > $ gdb -c /core /usr/bin/syslog-ng > > (gdb) thread apply all bt full > > > > This should give us a few hints. > > > > -- > > |8] > > > > I installed a newer version of the PCRE library. That seems to have worked > for now. If it happens again I will be sure to follow your instructions, > thanks. > Well, installing another version of the PCRE library did not solve my problem. syslog-ng is still segfaulting. I tried upgrading to 3.2.2 but that did nothing, so I tried to create the backtrace like you said. However I don't think it was helpful in my case. The following is the output of the backtrace: root@ip-10-34-150-77:/var/lib/syslog-ng# gdb -c core /usr/sbin/syslog-ng GNU gdb (GDB) 7.2-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/syslog-ng...done. [New Thread 14680] warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /usr/lib/libgthread-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgthread-2.0.so.0 Reading symbols from /lib/libglib-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libglib-2.0.so.0 Reading symbols from /usr/lib/libevtlog.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libevtlog.so.0 Reading symbols from /lib/libssl.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/libssl.so.0.9.8 Reading symbols from /lib/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.0.9.8 Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /usr/lib/libnet.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libnet.so.1 Reading symbols from /lib/libwrap.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libwrap.so.0 Reading symbols from /usr/lib/libdbi.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libdbi.so.0 Reading symbols from /lib/libcap.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcap.so.2 Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/libpcre.so.3 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libattr.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libattr.so.1 Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_compat.so.2 Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_nis.so.2 Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_dns.so.2 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Core was generated by `/usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --no-caps --enable-core'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd8440c4e9d in ?? () from /lib/libpcre.so.3 (gdb) thread apply all bt full Thread 1 (Thread 14680): #0 0x00007fd8440c4e9d in ?? () from /lib/libpcre.so.3 No symbol table info available. Cannot access memory at address 0x7fff4a4e3fd8 (gdb) If it is in anyway helpful, I think the only usage of PCRE in my syslog-ng.conf is a rewrite rule to change the Unicode character separator line to a new line: subst("\\p{Zl}", "\n", value("MESSAGE") type("pcre") flags("utf8" "global")); Is it related to my problem in some way?

2 1

[PATCH] afmongodb: Only free the value pairs if there's any.
by Gergely Nagy 16 Jan '12

16 Jan '12

In afmongodb_dd_free() only free self->vp, if it is non-NULL. In case of a syntax error in the configuration file, it may very well be NULL still, and that would lead to us segfaulting shortly after printing the error message. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index cbd0d16..7009b33 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -495,7 +495,8 @@ afmongodb_dd_free(LogPipe *d) g_free(self->user); g_free(self->password); g_free(self->host); - value_pairs_free(self->vp); + if (self->vp) + value_pairs_free(self->vp); log_dest_driver_free(d); } -- 1.7.7.3

2 1

[PATCH] afmongodb: Fix a race condition in the queue handling.
by Gergely Nagy 16 Jan '12

16 Jan '12

There is a race condition between afmongodb_dd_queue() and afmongodb_worker_insert(), that can trigger an assertion when the rate of incoming messages is very high. The scenario that triggers this is like this: * We receive a message, put it in our internal queue, and notify the worker thread. * The worker thread resets the queue's parallel_push. * We receive another message, notice that the queue is empty, so we insert it, and set parallel_push at the same time. * The worker thread tries to pop the queue's head, but that triggers an assert, because parallel_push is set. The root cause of the problem is that afmongodb_dd_queue() modifies the queue without locking the queue_mutex. This patch corrects that problem. Reported-By: Costa Farber <costaf(a)wix.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index cbd0d16..a88aeaa 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -530,7 +530,9 @@ afmongodb_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_optio g_mutex_lock(self->suspend_mutex); if (queue_was_empty && !self->writer_thread_suspended) { + g_mutex_lock(self->queue_mutex); log_queue_set_parallel_push(self->queue, 1, afmongodb_dd_queue_notify, self, NULL); + g_mutex_unlock(self->queue_mutex); } g_mutex_unlock(self->suspend_mutex); } -- 1.7.7.3

2 1

limit of "log_fifo_size"
by 外川　康徳 16 Jan '12

16 Jan '12

Hello folks, Is there limit of "log_fifo_size" on syslog-ng output message quing? I do know that real limit size of handling log messages depends on network environment or machine spec, but I'd like to know it has limit figure on its specification. Regards, Yasunori Togawa

2 1

set tag in log statement
by Patrick Hemmer 10 Jan '12

10 Jan '12

Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s. I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`. Such a config would look like filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); tags("tag1"); }; filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); tags("tag1"); }; filter f_c { not tags("tag1"); }; log { source(s_local); filter(f_c); destination (d_c); };

4 8

program destination and init??
by Matt Zagrabelny 09 Jan '12

09 Jan '12

Hi, I've got a question regarding the program destination and how/when it spawns the program. I've got a few program destination directives in my syslog-ng config. When I start the syslog-ng daemon, the program's (of the program destinations) parents are the syslog-ng daemon. After some amount of time (perhaps 20 or 30 hours) I get more programs spawned. Some of the programs have parents that are 'init' (PID = 1). One of my programs is 'vpndb' - see the grep output for an example: $ ps -ef | grep vpndb root 610 1 41 Nov23 ? 2-01:29:55 perl /usr/bin/vpndb root 4036 605 0 06:44 ? 00:00:00 /bin/sh -c /usr/bin/vpndb root 4039 4036 0 06:44 ? 00:00:00 perl /usr/bin/vpndb root 5231 1 26 Nov24 ? 1-02:09:12 perl /usr/bin/vpndb root 5322 1 1 Nov27 ? 00:21:26 perl /usr/bin/vpndb root 5658 1381 0 08:33 pts/4 00:00:00 grep vpndb root 16167 1 12 Nov26 ? 06:19:10 perl /usr/bin/vpndb root 26542 1 19 Nov25 ? 14:12:19 perl /usr/bin/vpndb Above there are 5 vpndb programs with a parent of PID = 1. One of the vpndb's belongs to syslog-ng. Two of my other program destinations are "ssh" commands. They exhibit the same symptoms (multiple programs running with parents of PID = 1). Hence I don't think this problem is related to the specific program being executed. Does anyone have any idea why I am getting multiple programs spawned? (I presume that syslog-ng is spawning these programs.) I do have cron jobs that will 'reload' the syslog-ng daemon. Thanks for any hints. -matt zagrabelny

5 11

[PATCH] logrewrite: Correctly initialize the conditional rules.
by Gergely Nagy 09 Jan '12

09 Jan '12

In case where the condition of a rewrite used a filter reference instead of inlining the filter itself, the filter expression was never initialized. This resulted in the condition always evaluating to not-match, making the following construct completely useless: rewrite("new-value", value("MESSAGE"), condition(filter(f_my_filter))) The fix is to introduce a new function, log_rewrite_init_method(), which will initialize the condition filter if it exists, and has an init method. Then we bind this method to LogPipe's init callback during log_rewrite_init(). This way, when initializing the rewrite pipe, the condition will get initialised too. Reported-by: Thomas Wollner <tw(a)wollner-net.de> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/logrewrite.c | 12 ++++++++++++ 1 files changed, 12 insertions(+), 0 deletions(-) diff --git a/lib/logrewrite.c b/lib/logrewrite.c index a578439..3ff245a 100644 --- a/lib/logrewrite.c +++ b/lib/logrewrite.c @@ -75,6 +75,17 @@ log_rewrite_free_method(LogPipe *s) log_process_pipe_free_method(s); } +static gboolean +log_rewrite_init_method(LogPipe *s) +{ + LogRewrite *self = (LogRewrite *) s; + + if (self->condition && self->condition->init) + self->condition->init(self->condition, log_pipe_get_config(s)); + + return TRUE; +} + static void log_rewrite_init(LogRewrite *self) { @@ -83,6 +94,7 @@ log_rewrite_init(LogRewrite *self) self->super.super.flags |= PIF_CLONE; self->super.super.free_fn = log_rewrite_free_method; self->super.super.queue = log_rewrite_queue; + self->super.super.init = log_rewrite_init_method; self->value_handle = LM_V_MESSAGE; } -- 1.7.7.3

2 1

Re: [syslog-ng] syslog-ng Digest, Vol 80, Issue 33
by Anup Shetty 23 Dec '11

23 Dec '11

I cant execute those commands. Here's the error Unknown command Syntax: pdbtool <command> [options] Possible commands are: match Match a message against the pattern database dump Dump pattern datebase tree merge Merge pattern databases dictionary Dump pattern dictionary Version syslog-ng-premium-edition 3.2.1 Installer-Version: 3.2.1 > > > > > > ------------------------------ > > Message: 3 > Date: Thu, 22 Dec 2011 13:11:05 -0600 > From: Martin Holste <mcholste(a)gmail.com> > Subject: Re: [syslog-ng] Pattern matching. > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Message-ID: > <CANpnLHgau7bZrSP2ro0QY=a8ZcJZLyqJgAVegWufDuszOjuCMA(a)mail.gmail.com > > > Content-Type: text/plain; charset=ISO-8859-1 > > You can also include an example pattern as part of the actual rule like > this: > > <ruleset> > <program></program> > <rule id="2"> > <pattern>@ESTRING:user::@ Security Microsoft > Windows security auditing.: [Success Audit] A computer account was > changed. Subject: Security ID: S-1-5-7 Account Name: > ANONYMOUS LOGON Account Domain: NT AUTHORITY Logon ID: 0x3e6 > Computer Account That Was Changed: Security ID: @ESTRING:: > @Account Name: @ESTRING:ACC_NAME: @ Account Domain: WW002 > Changed Attributes: SAM Account Name: - Display Name: - User > Principal Name: - Home Directory: - Home Drive: - Script Path: > - Profile Path: - User Workstations: - Password Last Set: > @ESTRING:: @@ESTRING:: @ Account Expires: - Primary Group ID: - > AllowedToDelegateTo: - Old UAC Value: - New UAC Value: - User > Account Control: - User Parameters: - SID History: - Logon > Hours: - DNS Host Name: - Service Principal Names: - > Additional Information: Privileges: - (EventID 4742)</pattern> > <examples> > <example> > <test_message > program="Microsoft_Windows_security_auditing.[5784]">: Security > Microsoft Windows security auditing.: [Success Audit] A computer > account was changed. Subject: Security ID: S-1-5-7 Account > Name: ANONYMOUS LOGON Account Domain: NT AUTHORITY Logon ID: > 0x3e6 Computer Account That Was Changed: Security ID: > S-1-5-21-776561741-789336058-725345543-305444 Account Name: User1$ > Account Domain: TEST Changed Attributes: SAM Account Name: - > Display Name: - User Principal Name: - Home Directory: - Home > Drive: - Script Path: - Profile Path: - User Workstations: - > Password Last Set: 12/22/2011 3:38:32 AM Account Expires: - > Primary Group ID: - AllowedToDelegateTo: - Old UAC Value: - New > UAC Value: - User Account Control: - User Parameters: - SID > History: - Logon Hours: - DNS Host Name: - Service Principal > Names: - Additional Information: Privileges: - (EventID > 4742)</test_message> > <test_value > name="ACC_NAME">User1$</test_value> > </example> > </examples> > </rule> > </ruleset> > > Then you can test it more easily like this: > pdbtool test patterndb.xml > > On Thu, Dec 22, 2011 at 8:04 AM, Balazs Scheidler <bazsi(a)balabit.hu> > wrote: > > On Thu, 2011-12-22 at 14:31 +0530, Anup Shetty wrote: > >> Nope, no luck yet. Still blanks being spit out. > >> > >> > >> Here's the exact extract of the pattern matching and the log: > >> > >> > >> Pattern String > >> --------------------------- > >> > >> > >> @ESTRING:user::@ Security Microsoft Windows security auditing.: > >> [Success Audit] A computer account was changed. ? ?Subject: ? Security > >> ID: ?S-1-5-7 ? Account Name: ?ANONYMOUS LOGON ? Account Domain: ?NT > >> AUTHORITY ? Logon ID: ?0x3e6 ? ?Computer Account That Was Changed: > >> Security ID: ?@ESTRING:: ?@Account Name: ? @ESTRING:ACC_NAME: @ > >> Account Domain: ?WW002 ? ?Changed Attributes: ? SAM Account Name: - > >> Display Name: ?- ? User Principal Name: - ? Home Directory: ?- ? Home > >> Drive: ?- ? Script Path: ?- ? Profile Path: ?- ? User Workstations: - > >> Password Last Set: @ESTRING:: @@ESTRING:: @ ? Account Expires: ?- > >> Primary Group ID: - ? AllowedToDelegateTo: - ? Old UAC Value: ?- ? New > >> UAC Value: ?- ? User Account Control: - ? User Parameters: - ? SID > >> History: ?- ? Logon Hours: ?- ? DNS Host Name: ?- ? Service Principal > >> Names: - ? ?Additional Information: ? Privileges: ?- (EventID 4742) > >> > >> > >> Log > >> ------------------ > >> > >> > >> Dec 22 03:38:32 Server.zoom11.test.net > >> Microsoft_Windows_security_auditing.[5784]: : Security Microsoft > >> Windows security auditing.: [Success Audit] A computer account was > >> changed. ? ?Subject: ? Security ID: ?S-1-5-7 ? Account Name: > >> ?ANONYMOUS LOGON ? Account Domain: ?NT AUTHORITY ? Logon ID: ?0x3e6 > >> ?Computer Account That Was Changed: ? Security ID: > >> ?S-1-5-21-776561741-789336058-725345543-305444 ? Account Name: ?User1$ > >> Account Domain: ?TEST ? ?Changed Attributes: ? SAM Account Name: - > >> Display Name: ?- ? User Principal Name: - ? Home Directory: ?- ? Home > >> Drive: ?- ? Script Path: ?- ? Profile Path: ?- ? User Workstations: - > >> Password Last Set: 12/22/2011 3:38:32 AM ? Account Expires: ?- > >> Primary Group ID: - ? AllowedToDelegateTo: - ? Old UAC Value: ?- ? New > >> UAC Value: ?- ? User Account Control: - ? User Parameters: - ? SID > >> History: ?- ? Logon Hours: ?- ? DNS Host Name: ?- ? Service Principal > >> Names: - ? ?Additional Information: ? Privileges: ?- (EventID 4742) > >> > >> > > "pdbtool match" can be used to test patterns. > > > > pdbtool patch -p <path to xml file> -P '<appname>' -M '<msg>' --debug > --color-out > > > > This even colours the output so that the partial matches can be > > recognized. This is the best way to troubleshoot patterns. > > > > -- > > Bazsi > > > > > -- Thanks and regards, Anup

2 1