- syslog-ng - lists.balabit.hu

[PATCH (3.4) 0/5] afmongodb cleanups & replica set support
by Gergely Nagy 16 Jan '12

16 Jan '12

Following this email, a few patches will come, various minor touch-ups (such as the cleanup of the default value-pairs config, and the removal of the TODO file) and replica-set support for afmongodb. The replica-set support is an incompatible change compared to 3.3, as it removes the host() and port() options from the driver, replacing both with a servers() option. After the patches, the driver will support replica sets, which means that when connecting to a replica set, and the primary goes down, the driver will reconnect to the new master automatically. Most of this is handled transparently by the underlying libmongo-client library, so the syslog-ng side doesn't actually get much more complicated. These patches have been sitting in my syslog-ng module collection repo since about july, and I kinda forgot about them. It's about time I submit them. (And consequently, the module-collection repo will be abandoned, as it pretty much has been for the past few months anyway. Turns out it's easier for me to maintain module updates within my syslog-ng repo, especially now that I have 3.3 and 3.4 within the same git repo.) All of these are available from the feature/3.4/afmongodb/lmc-sync branch of my git repository.

2 7

Syslog-ng 3.2 connection timeout with firewall
by André Larose 16 Jan '12

16 Jan '12

Hi, I have two syslog-ng 3.2, one client and one server. The two are separated by a firewall and a load balancer. I noticed that after some inactivity I was not able to receive logs from my client. So I started some tcpdump on both servers to check the traffic. From what I see the firewall will close the connection after some time, so when the client sends traffic it gets dropped. I added the keep-alive(yes) and so_keep-alive(yes) and the mark_freq(60) to the configs. But I still do not see keepalive packets with tcpdump. Am I missing other parameters to have "keepalive" traffic sent ? Thank you in advance. ############################################################################# # # Configuration file syslog-ng.conf.client # @version: 3.2 @include "scl.conf" # add global settings options { # Disable DNS usage to gain the process time use_dns(no); flush_lines(1); flush_timeout(500); mark(59); }; source s_TCP { tcp( port(555) flags(no-parse) ); }; source s_syslogng { internal(); }; destination d_syslogng { file("/opt/syslog-ng/logs/$YEAR$MONTH$DAY.syslog-ng.log"); }; destination d_TCP { tcp ( "server1" port(556) keep-alive(yes) so_keepalive(yes) flags(no_multi_line) flush_lines(1) flush_timeout(500) log_fifo_size(4096) template("$MSG\n") template_escape(no) ); }; .... ############################################################################# # # Configuration file syslog-ng.conf.server # @version: 3.2 @include "scl.conf" # add global settings options { # Disable DNS usage to gain the process time use_dns(no); flush_lines(1); flush_timeout(500); mark(59); }; source s_TCP_555 { tcp( port(555) flags(no-parse) max-connections(200) use_dns(no) ); }; source s_TCP_556 { tcp( port(556) flags(no-parse) so_keepalive(yes) use_dns(no) ); }; source s_syslogng { internal(); }; destination d_syslogng { file("/opt/syslog-ng/logs/$YEAR$MONTH$DAY.syslog-ng.log"); }; destination d_local { file("/opt2/syslog-ng/logs/$YEAR$MONTH$DAY.TNT.Messages.log" log_fifo_size(8192) template("$MSG\n") template_escape(no) ); }; ...

2 1

[PATCH (3.4)] center: Fix a few syntax errors.
by Gergely Nagy 16 Jan '12

16 Jan '12

Fix two syntax errors (an unclosed comment, and a double-closing-bracer) that came in with a previous patch. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/center.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/lib/center.c b/lib/center.c index 986ab75..2fc3554 100644 --- a/lib/center.c +++ b/lib/center.c @@ -320,6 +320,7 @@ log_center_instantiate_process_pipe_line(LogCenter *self, LogProcessRule *rule) * * The next member pointer is not holding a reference, but can be * assumed to be kept alive as long as the configuration is running. + */ LogPipe * log_center_init_pipe_line(LogCenter *self, LogConnection *conn, GlobalConfig *cfg, gboolean toplevel, gboolean flow_controlled_parent) { @@ -468,7 +469,6 @@ log_center_init_pipe_line(LogCenter *self, LogConnection *conn, GlobalConfig *cf log_multiplexer_add_next_hop(fork_mpx, sub_pipe); break; } - } default: { g_assert_not_reached(); -- 1.7.7.3

2 1

Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu 10.10 (Luis Pugoy)
by Luis Pugoy 16 Jan '12

16 Jan '12

> > > Message: 1 > Date: Mon, 5 Dec 2011 19:54:42 +0800 > From: Luis Pugoy <lpugoy(a)insynchq.com> > Subject: Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu > 10.10 > To: syslog-ng(a)lists.balabit.hu > Message-ID: > <CA+WGxEhKYMQXbYTVUZVScjaXDdjRETGSCk5nZaisW9p-UOQWoA(a)mail.gmail.com > > > Content-Type: text/plain; charset="iso-8859-1" > > > > > Message: 3 > > Date: Mon, 05 Dec 2011 10:10:40 +0100 > > From: Gergely Nagy <algernon(a)balabit.hu> > > Subject: Re: [syslog-ng] syslog-ng 3.1.2 segmentation fault in Ubuntu > > 10.10 > > To: Syslog-ng users' and developers' mailing list > > <syslog-ng(a)lists.balabit.hu> > > Message-ID: <87aa77l6y7.fsf(a)algernon.balabit> > > Content-Type: text/plain > > > > Luis Pugoy <lpugoy(a)insynchq.com> writes: > > > > > Does anyone have any insight into this? The segfault is happening more > > > often now. :( > > > > I'd suggest either upgrading to 3.3 from my repo[1], or if an upgrade is > > not feasible, then if you could compile 3.1 with debug symbols, get it > > to drop a core, and do a backtrace, that would help tremendously. > > > > [1]: http://asylum.madhouse-project.org/projects/debian/ > > > > To do the latter, the following steps should work: > > > > $ apt-get source syslog-ng > > $ sudo apt-get build-dep syslog-ng > > $ cd syslog-ng-* > > $ DEB_BUILD_OPTIONS="debug nostrip" dpkg-buildpackage -us -uc -rfakeroot > > $ sudo dpkg -i ../syslog-ng_*.deb > > > > Then add --enable-core to SYSLOGNG_OPTS in /etc/default/syslog-ng, and > > when it segfaults, it'll make a core in (as far as I remember) /. Once > > that happened, do something like this: > > > > $ gdb -c /core /usr/bin/syslog-ng > > (gdb) thread apply all bt full > > > > This should give us a few hints. > > > > -- > > |8] > > > > I installed a newer version of the PCRE library. That seems to have worked > for now. If it happens again I will be sure to follow your instructions, > thanks. > Well, installing another version of the PCRE library did not solve my problem. syslog-ng is still segfaulting. I tried upgrading to 3.2.2 but that did nothing, so I tried to create the backtrace like you said. However I don't think it was helpful in my case. The following is the output of the backtrace: root@ip-10-34-150-77:/var/lib/syslog-ng# gdb -c core /usr/sbin/syslog-ng GNU gdb (GDB) 7.2-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/syslog-ng...done. [New Thread 14680] warning: Can't read pathname for load map: Input/output error. Reading symbols from /lib/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/librt.so.1 Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libnsl.so.1 Reading symbols from /usr/lib/libgthread-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libgthread-2.0.so.0 Reading symbols from /lib/libglib-2.0.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libglib-2.0.so.0 Reading symbols from /usr/lib/libevtlog.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libevtlog.so.0 Reading symbols from /lib/libssl.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/libssl.so.0.9.8 Reading symbols from /lib/libcrypto.so.0.9.8...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypto.so.0.9.8 Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /usr/lib/libnet.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libnet.so.1 Reading symbols from /lib/libwrap.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libwrap.so.0 Reading symbols from /usr/lib/libdbi.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libdbi.so.0 Reading symbols from /lib/libcap.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libcap.so.2 Reading symbols from /lib/libpcre.so.3...(no debugging symbols found)...done. Loaded symbols for /lib/libpcre.so.3 Reading symbols from /lib/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpthread.so.0 Reading symbols from /lib/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/libm.so.6 Reading symbols from /lib/libattr.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libattr.so.1 Reading symbols from /lib/libnss_compat.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_compat.so.2 Reading symbols from /lib/libnss_nis.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_nis.so.2 Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_dns.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_dns.so.2 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Core was generated by `/usr/sbin/syslog-ng -p /var/run/syslog-ng.pid --no-caps --enable-core'. Program terminated with signal 11, Segmentation fault. #0 0x00007fd8440c4e9d in ?? () from /lib/libpcre.so.3 (gdb) thread apply all bt full Thread 1 (Thread 14680): #0 0x00007fd8440c4e9d in ?? () from /lib/libpcre.so.3 No symbol table info available. Cannot access memory at address 0x7fff4a4e3fd8 (gdb) If it is in anyway helpful, I think the only usage of PCRE in my syslog-ng.conf is a rewrite rule to change the Unicode character separator line to a new line: subst("\\p{Zl}", "\n", value("MESSAGE") type("pcre") flags("utf8" "global")); Is it related to my problem in some way?

2 1

[PATCH] afmongodb: Only free the value pairs if there's any.
by Gergely Nagy 16 Jan '12

16 Jan '12

In afmongodb_dd_free() only free self->vp, if it is non-NULL. In case of a syntax error in the configuration file, it may very well be NULL still, and that would lead to us segfaulting shortly after printing the error message. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index cbd0d16..7009b33 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -495,7 +495,8 @@ afmongodb_dd_free(LogPipe *d) g_free(self->user); g_free(self->password); g_free(self->host); - value_pairs_free(self->vp); + if (self->vp) + value_pairs_free(self->vp); log_dest_driver_free(d); } -- 1.7.7.3

2 1

[PATCH] afmongodb: Fix a race condition in the queue handling.
by Gergely Nagy 16 Jan '12

16 Jan '12

There is a race condition between afmongodb_dd_queue() and afmongodb_worker_insert(), that can trigger an assertion when the rate of incoming messages is very high. The scenario that triggers this is like this: * We receive a message, put it in our internal queue, and notify the worker thread. * The worker thread resets the queue's parallel_push. * We receive another message, notice that the queue is empty, so we insert it, and set parallel_push at the same time. * The worker thread tries to pop the queue's head, but that triggers an assert, because parallel_push is set. The root cause of the problem is that afmongodb_dd_queue() modifies the queue without locking the queue_mutex. This patch corrects that problem. Reported-By: Costa Farber <costaf(a)wix.com> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index cbd0d16..a88aeaa 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -530,7 +530,9 @@ afmongodb_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_optio g_mutex_lock(self->suspend_mutex); if (queue_was_empty && !self->writer_thread_suspended) { + g_mutex_lock(self->queue_mutex); log_queue_set_parallel_push(self->queue, 1, afmongodb_dd_queue_notify, self, NULL); + g_mutex_unlock(self->queue_mutex); } g_mutex_unlock(self->suspend_mutex); } -- 1.7.7.3

2 1

limit of "log_fifo_size"
by 外川　康徳 16 Jan '12

16 Jan '12

Hello folks, Is there limit of "log_fifo_size" on syslog-ng output message quing? I do know that real limit size of handling log messages depends on network environment or machine spec, but I'd like to know it has limit figure on its specification. Regards, Yasunori Togawa

2 1

Antw: Re: v3.3 not flushing lines?
by Peter Eisenlohr 14 Jan '12

14 Jan '12

Hi, just tried to reproduce with a simple perl script: open FILE, ">/home/peter/test.log" or die $!; { my $ofh = select FILE; $| = 1; select $ofh; } while (<>) { chomp; print FILE "incoming: $_\n"; }; Now, when sending log input to syslog-ng via tcp(), each line I type immediatelly shows up in the output log file. => can not reproduce. That is, with version 3.3.1. Peter Dipl.-Inf. Peter Eisenlohr Geschäftsbereich Logistik INFORM GmbH, Pascalstr. 23, 52076 Aachen, Germany Telefon: (+49) 2408 9456-6000 E-Mail: peter.eisenlohr(a)inform-software.com, Web: http://www.inform-software.com INFORM Institut fur Operations Research und Management GmbH Registered AmtsG Aachen HRB1144 Gfhr. Adrian Weiler >>> Clayton Dukes destination d_logzilla { 13.01.12 16.03 Uhr >>> program("/var/www/logzilla/scripts/db_insert.pl" template("$R_YEAR-$R_MONTH-$R_DAY $R_HOUR:$R_MIN:$S_SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n") template_escape(yes) );

2 1

v3.3 not flushing lines?
by Clayton Dukes 13 Jan '12

13 Jan '12

Hey folks, I have a user experiencing an issue where some of the incoming logs are not getting piped to my perl script until the second time we generate events. I believe this is generally set using flush_lines(1), but it doesn’t seem to be honoring that setting in the config. I also tried adding flush_timeout(900), but that has no affect. In the output below, if I quit the telnet and start it again, the previously missing events are then received (but the current ones are missing) - basically, it doesn't appear to be flushing every single line. Possible other reason: has something changed between v2.x and 3.x where the program() destination would possibly not send an EOF signal - i.e. is the pipe now kept open vs. an individual call to the program each time in v2.x? To verify that this is happening: Term 1: tail -f /tmp/logzilla_import.txt Term 2: /usr/local/sbin/syslog-ng -f /etc/syslog-ng/syslog-ng.conf -Fdv Term 3: telnet 192.168.254.1 Term 1 results: 192.168.254.1 22 7 3732620769 .Jan 11 2012 19:29:02.284 EST: Telnet2: 1 1 251 1 3751981041 0 2012-01-11 19:29:02 2012-01-11 19:29:02 192.168.254.1 22 7 3732620769 .Jan 11 2012 19:29:02.284 EST: TCP2: Telnet sent WILL ECHO (1) 3751981041 0 2012-01-11 19:29:02 2012-01-11 19:29:02 Term 2 results: Incoming log entry; line='<183>6987: .Jan 11 2012 19:29:02.284 EST: TCP2: Telnet sent WILL ECHO (1)' Incoming log entry; line='<183>6988: .Jan 11 2012 19:29:02.284 EST: Telnet2: 2 2 251 3' Incoming log entry; line='<183>6989: .Jan 11 2012 19:29:02.284 EST: TCP2: Telnet sent WILL SUPPRESS-GA (3)' Incoming log entry; line='<183>6990: .Jan 11 2012 19:29:02.284 EST: Telnet2: 80000 80000 253 24' Incoming log entry; line='<183>6991: .Jan 11 2012 19:29:02.284 EST: TCP2: Telnet sent DO TTY-TYPE (24)' Incoming log entry; line='<183>6992: .Jan 11 2012 19:29:02.284 EST: Telnet2: 10000000 10000000 253 31' Incoming log entry; line='<183>6993: .Jan 11 2012 19:29:02.284 EST: TCP2: Telnet sent DO WINDOW-SIZE (31)' Incoming log entry; line='<183>6994: .Jan 11 2012 19:29:02.284 EST: TCP2: Telnet received DO ENCRYPTION (38)' Incoming log entry; line='<183>6995: .Jan 11 2012 19:29:02.284 EST: TCP2: Telnet sent WONT ENCRYPTION (38) (unimplemented)' Incoming log entry; line='<183>6996: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received WILL ENCRYPTION (38)' Incoming log entry; line='<183>6997: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet sent DONT ENCRYPTION (38) (unimplemented)' Incoming log entry; line='<183>6998: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received DO SUPPRESS-GA (3)' Incoming log entry; line='<183>6999: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received WILL TTY-TYPE (24)' Incoming log entry; line='<183>7000: .Jan 11 2012 19:29:02.292 EST: Telnet2: Sent SB 24 1 ' Incoming log entry; line='<183>7001: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received WILL WINDOW-SIZE (31)' Incoming log entry; line='<183>7002: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received WILL TTY-SPEED (32) (refused)' Incoming log entry; line='<183>7003: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet sent DONT TTY-SPEED (32)' Incoming log entry; line='<183>7004: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received WILL LOCAL-FLOW (33) (refused)' Incoming log entry; line='<183>7005: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet sent DONT LOCAL-FLOW (33)' Incoming log entry; line='<183>7006: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received WILL LINEMODE (34)' Incoming log entry; line='<183>7007: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet sent DONT LINEMODE (34) (unimplemented)' Incoming log entry; line='<183>7008: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received WILL NEW-ENVIRON (39)' Incoming log entry; line='<183>7009: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet sent DONT NEW-ENVIRON (39) (unimplemented)' Incoming log entry; line='<183>7010: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received DO STATUS (5)' Incoming log entry; line='<183>7011: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet sent WONT STATUS (5) (unimplemented)' Incoming log entry; line='<183>7012: .Jan 11 2012 19:29:02.292 EST: TCP2: Telnet received DO ECHO (1)' Incoming log entry; line='<183>7013: .Jan 11 2012 19:29:02.292 EST: Telnet2: recv SB NAWS 132 63' Incoming log entry; line='<183>7014: .Jan 11 2012 19:29:02.292 EST: Telnet2: recv SB 24 0 LINUX' Incoming log entry; line='<183>7015: .Jan 11 2012 19:29:02.493 EST: TCP2: Telnet received WILL ENVIRONMENT (36) (refused)' Incoming log entry; line='<183>7016: .Jan 11 2012 19:29:02.493 EST: TCP2: Telnet sent DONT ENVIRONMENT (36)' ______________________________________________________________ Clayton Dukes ______________________________________________________________

3 5

syslog-ng Insider - January 2012
by Peter Czanik 12 Jan '12

12 Jan '12

Dear syslog-ng users, This is the 10th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> POLL We would like to know, which features of syslog-ng do you regard as most important. Please check our survey at http://balabit.com/poll/syslog-ng-features FEATURED NEWS syslog-ng 3.3.4 is being prepared --------------------------------- The release of 3.3.3 brought many new users to the latest syslog-ng version, which helped to uncover some more bugs in advanced configurations. Until a new release arrives, please check the git tree ( git://git.balabit.hu/bazsi/syslog-ng-3.3 ) and the mailing list archives, if your problem is addressed by a patch. If working directly from git is problematic in your environment, a daily shapshot of 3.3 development is available at http://packages.madhouse-project.org/syslog-ng/3.3/syslog-ng-3.3-HEAD.tar.gz Meet us at FOSDEM 2012 in Brussels! ----------------------------------- There will be one or two presentations held by BalaBit guys this year. As schedule and rooms are not yet final, please check http://www.fosdem.org/ if you want to meet us or drop me an e-mail! Junction -------- Version 3.4 of syslog-ng is under intensive development. One of the new features, the “junction”, which makes syslog-ng.conf even more flexible, was just introduced on the syslog-ng mailing list: https://lists.balabit.hu/pipermail/syslog-ng/2012-January/018074.html OTHER SHORT NEWS * syslog-ng web GUI blog updated: http://czanik.blogs.balabit.com/2011/12/graphical-user-interfaces-for-use-w… WHITE PAPERS * SIEM solutions are the core of enterprise security. But they are not much use without data integrity and high performance centralized log collection. Learn how to enhance your SIEM with syslog-ng from our latest white paper at http://www.balabit.com/support/documentation/uc_opt_siem_en_web03.pdf ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

documentation missing sql options
by Patrick Hemmer 10 Jan '12

10 Jan '12

The documentation (at least http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid… <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid…>) is missing a couple of options for the sql destination driver * session-statements * retries Was just going through and noticed cause I need to use the session-statements one. <http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guid…>

1 1

set tag in log statement
by Patrick Hemmer 10 Jan '12

10 Jan '12

Would it be possible to set tags on a message when it is matched by a `log {}` statement? This would be useful for complex log configurations so you could filter out messages that were logged by previous `log {}` statements without having to write really long `filter {}`s. I'm not sure if this is possible or not. Would depend on the order of operations, like if a filter block in a latter log statement is evaluated at the same time as an earlier filter block, or if its evaluated when it reaches the `log {}`. Such a config would look like filter f_a { facility(mail); }; log { source(s_local); filter(f_a); destination(d_a); tags("tag1"); }; filter f_b { facility(local0); }; log { source(s_local); filter(f_b); destination(d_b); tags("tag1"); }; filter f_c { not tags("tag1"); }; log { source(s_local); filter(f_c); destination (d_c); };

4 8

Escape slashes etc in macro variables ($PROGRAM)
by Göran Sandahl 10 Jan '12

10 Jan '12

Howdy, Is there a option for escaping macro variables such as $PROGRAM? I've seen a couple of discussions about this on the list but they never seem to come to a conclusion. Case in point: I'm creating an output file macro like this: file("/var/log/$SOURCEIP.$PROGRAM.log" ... and Syslog-ng failes to write the file whenever $PROGRAM looks similar to "postfix/quemgr" since its then trying to "create" the directory postfix due to the slash. Of course, thats not what I want. Can this be circumvented somehow? Cheers! Goran

5 5

Re: [syslog-ng] Syslog-ng 3.2 connection timeout with firewall
by André Larose 10 Jan '12

10 Jan '12

Hi, I found the timeout issue. It turns out that the keep_alive is not generated by Syslog-NG but by the OS. I adjusted the following values: tcp_keepalive_time tcp_keepalive_intvl tcp_keepalive_probes First in the live Kernel with the command : sysctl -w net.ipv4.tcp_keepalive_time=180 net.ipv4.tcp_keepalive_intvl=180 net.ipv4.tcp_keepalive_probes=9 Then I made it permanent with adding to /etc/sysctl.conf: cat >>/etc/sysctl.conf <<EOF # # Keepalive parameters for the syslog-ng # net.ipv4.tcp_keepalive_time = 180 net.ipv4.tcp_keepalive_intvl = 180 net.ipv4.tcp_keepalive_probes = 9 EOF You can verify your work is done with the following command: sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes net.ipv4.tcp_keepalive_time = 180 net.ipv4.tcp_keepalive_intvl = 180 net.ipv4.tcp_keepalive_probes = 9 I found the information from this URL: http://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html Maybe it could be mentioned in the documentation that the so_keep-alive option can be tweaked at the kernel level. Regards, ------------------------------ Message: 8 Date: Tue, 3 Jan 2012 17:46:06 -0500 From: Andr? Larose <andre.larose(a)telus.com<mailto:andre.larose@telus.com>> Subject: [syslog-ng] Syslog-ng 3.2 connection timeout with firewall To: "syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>" <syslog-ng(a)lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Message-ID: <BDB004F99074254BBCEE3AB94F31FDF5435439F706(a)WP40066.corp.ads<mailto:BDB004F99074254BBCEE3AB94F31FDF5435439F706@WP40066.corp.ads>> Content-Type: text/plain; charset="iso-8859-1" Hi, I have two syslog-ng 3.2, one client and one server. The two are separated by a firewall and a load balancer. I noticed that after some inactivity I was not able to receive logs from my client. So I started some tcpdump on both servers to check the traffic. From what I see the firewall will close the connection after some time, so when the client sends traffic it gets dropped. I added the keep-alive(yes) and so_keep-alive(yes) and the mark_freq(60) to the configs. But I still do not see keepalive packets with tcpdump. Am I missing other parameters to have "keepalive" traffic sent ? Thank you in advance.

1 0

program destination and init??
by Matt Zagrabelny 09 Jan '12

09 Jan '12

Hi, I've got a question regarding the program destination and how/when it spawns the program. I've got a few program destination directives in my syslog-ng config. When I start the syslog-ng daemon, the program's (of the program destinations) parents are the syslog-ng daemon. After some amount of time (perhaps 20 or 30 hours) I get more programs spawned. Some of the programs have parents that are 'init' (PID = 1). One of my programs is 'vpndb' - see the grep output for an example: $ ps -ef | grep vpndb root 610 1 41 Nov23 ? 2-01:29:55 perl /usr/bin/vpndb root 4036 605 0 06:44 ? 00:00:00 /bin/sh -c /usr/bin/vpndb root 4039 4036 0 06:44 ? 00:00:00 perl /usr/bin/vpndb root 5231 1 26 Nov24 ? 1-02:09:12 perl /usr/bin/vpndb root 5322 1 1 Nov27 ? 00:21:26 perl /usr/bin/vpndb root 5658 1381 0 08:33 pts/4 00:00:00 grep vpndb root 16167 1 12 Nov26 ? 06:19:10 perl /usr/bin/vpndb root 26542 1 19 Nov25 ? 14:12:19 perl /usr/bin/vpndb Above there are 5 vpndb programs with a parent of PID = 1. One of the vpndb's belongs to syslog-ng. Two of my other program destinations are "ssh" commands. They exhibit the same symptoms (multiple programs running with parents of PID = 1). Hence I don't think this problem is related to the specific program being executed. Does anyone have any idea why I am getting multiple programs spawned? (I presume that syslog-ng is spawning these programs.) I do have cron jobs that will 'reload' the syslog-ng daemon. Thanks for any hints. -matt zagrabelny

5 11

[PATCH] logrewrite: Correctly initialize the conditional rules.
by Gergely Nagy 09 Jan '12

09 Jan '12

In case where the condition of a rewrite used a filter reference instead of inlining the filter itself, the filter expression was never initialized. This resulted in the condition always evaluating to not-match, making the following construct completely useless: rewrite("new-value", value("MESSAGE"), condition(filter(f_my_filter))) The fix is to introduce a new function, log_rewrite_init_method(), which will initialize the condition filter if it exists, and has an init method. Then we bind this method to LogPipe's init callback during log_rewrite_init(). This way, when initializing the rewrite pipe, the condition will get initialised too. Reported-by: Thomas Wollner <tw(a)wollner-net.de> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/logrewrite.c | 12 ++++++++++++ 1 files changed, 12 insertions(+), 0 deletions(-) diff --git a/lib/logrewrite.c b/lib/logrewrite.c index a578439..3ff245a 100644 --- a/lib/logrewrite.c +++ b/lib/logrewrite.c @@ -75,6 +75,17 @@ log_rewrite_free_method(LogPipe *s) log_process_pipe_free_method(s); } +static gboolean +log_rewrite_init_method(LogPipe *s) +{ + LogRewrite *self = (LogRewrite *) s; + + if (self->condition && self->condition->init) + self->condition->init(self->condition, log_pipe_get_config(s)); + + return TRUE; +} + static void log_rewrite_init(LogRewrite *self) { @@ -83,6 +94,7 @@ log_rewrite_init(LogRewrite *self) self->super.super.flags |= PIF_CLONE; self->super.super.free_fn = log_rewrite_free_method; self->super.super.queue = log_rewrite_queue; + self->super.super.init = log_rewrite_init_method; self->value_handle = LM_V_MESSAGE; } -- 1.7.7.3

2 1

Error while starting Syslog-ng
by Kareep Navas 03 Jan '12

03 Jan '12

Error creating AF_INET socket (Error 0) Error initializing configuration, exiting.

2 1

What is the latest version of Syslog-ng?
by Kareep Navas 03 Jan '12

03 Jan '12

Hi All, We are currently using syslog-ng 1.4.14 and i would like to use & know the latest version of this syslog-ng? What is the purpose of the latest version? How to download the latest version? What are all the dependencies? Please direct me.. Thank you,

2 1

missing IP information at the message part
by Oğuz Yarımtepe 01 Jan '12

01 Jan '12

I have syslog-ng and logzilla 2.9.9g installed on a Debian squeeze machine. I can see the logs at the web interface. I configured syslog-ng to collect apache error and access logs. The problem is that i don't see IP information at the message part, on web interface. For ex, one of the machine that is sending apache logs is "www" machine. The syslog-ng configuration is as below: # All messages send to a remote site # #log { source(s_src); destination(d_net); }; #manual edit options { log_msg_size(8192); }; ####### # sources # ####### source s_apache_access { # unix-stream("/var/log/httpd/apache_log.socket" # max-connections(512) # keep-alive(yes)); file ("/var/log/apache2/access.log" flags(no-parse)); }; source s_apache_error { # unix-stream("/var/log/httpd/apache_log.socket" # max-connections(512) # keep-alive(yes)); file ("/var/log/apache2/error.log" flags(no-parse)); }; ################ # destinations # ################ destination d_custom_access { file("/var/log/apache2/custom_access.log"); }; destination d_custom_error { file("/var/log/apache2/custom_error.log"); }; destination d_apache_tls { tcp("192.168.1.145" port(6514) tls(ca_dir("/etc/syslog-ng/etc/ca.d") key_file("/etc/syslog-ng/etc/key.d/debian4key.pem") cert_file("/etc/syslog-ng/etc/cert.d/debian4cert.pem"))); }; ########### # filters # ########### filter f_apache_access { message("GET|POST"); }; filter f_apache_error { message("error"); }; log { source(s_apache_access); # filter(f_apache_access); destination(d_apache_tls); }; log { source(s_apache_error); # filter(f_apache_access); destination(d_apache_tls); }; and # tail -n1 /var/log/apache2/access.log 78.185.240.170 - - [01/Jan/2012:13:35:06 +0200] "GET /images/rand/randpic11.jpg HTTP/1.1" 200 45670 "http://www.comu.edu.tr/" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.7 (KHTML, like Gecko) Chrome/16.0.912.63 Safari/535.7" www:/home/oguz# For the www machine i don't see the 78.185.240.170 information at the web ui, but the rest of the line starting with -- Thogh another machine, "reverse", is sending apache logs with and additional field and i can see its log at the web ui without any missing information. The reverse has the same configuration except from the key information. # tail -n1 /var/log/apache2/other_vhosts_access.log www.beeseurope.eu:80 178.154.160.29 - - [01/Jan/2012:13:36:46 +0200] "GET /mod/forum/discuss.php?d=55&parent=88 HTTP/1.1" 200 34557 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" At the web interface, I can see the IP information also at the message part. My syslog-ng configuration at the server is as below: options { long_hostnames(off); # doesn't actually help on Solaris, log(3) truncates at 1024 chars log_msg_size(8192); # buffer just a little for performance flush_lines(16384); # memory is cheap, buffer messages unable to write (like to loghost) log_fifo_size(16384); # Hosts we don't want syslog from #bad_hostname("^(ctld.|cmd|tmd|last)$"); # The time to wait before a dead connection is reestablished (seconds) time_reopen(10); #Use DNS so that our good names are used, not hostnames use_dns(yes); dns_cache(yes); #Use the whole DNS name use_fqdn(yes); keep_hostname(yes); chain_hostnames(no); #Read permission for everyone perm(0644); # The default action of syslog-ng 1.6.0 is to log a STATS line # to the file every 10 minutes. That's pretty ugly after a while. # Change it to every 12 hours so you get a nice daily update of # # how many messages syslog-ng missed (0). # stats(43200); log_msg_size(8192); }; source s_apache { tcp(ip(0.0.0.0) port(6514) tls( key_file("/etc/syslog-ng/etc/key.d/debian1key.pem") cert_file("/etc/syslog-ng/etc/cert.d/debian1cert.pem") ca_dir("/etc/syslog-ng/etc/ca.d"))); }; filter f_apache_access { message("GET|POST"); }; filter f_apache_error { message("error"); }; # Create destination to LogZilla destination d_logzilla { program("/opt/logzilla-2.9.9g/scripts/db_insert.pl" template("$HOST\t$FACILITY\t$PRIORITY\t$LEVEL\t$TAG\t$YEAR-$MONTH-$DAY\t$HOUR:$MIN:$SEC\t$PROGRAM\t$MSG\n") #template("$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC\t$HOST\t$PRI\t$PROGRAM\t$MSGONLY\n") ); }; # test purposes destination d_apache_test_access { file("/var/log/apache2/hosts/$HOST/$YEAR/$MONTH/$DAY/access.log" create_dirs(yes)); }; destination d_apache_test_error { file("/var/log/apache2/hosts/$HOST/$YEAR/$MONTH/$DAY/error.log" create_dirs(yes)); }; # Tell syslog-ng to log to our new destination log { source(s_apache); destination(d_logzilla); }; log { source(s_apache); filter(f_apache_error); destination(d_apache_test_error); }; log { source(s_apache); filter(f_apache_access); destination(d_apache_test_access); }; Any ides about how i can fix the missing IP information at the web ui of logzilla?

1 0

Re: [syslog-ng] syslog-ng Digest, Vol 80, Issue 33
by Anup Shetty 23 Dec '11

23 Dec '11

I cant execute those commands. Here's the error Unknown command Syntax: pdbtool <command> [options] Possible commands are: match Match a message against the pattern database dump Dump pattern datebase tree merge Merge pattern databases dictionary Dump pattern dictionary Version syslog-ng-premium-edition 3.2.1 Installer-Version: 3.2.1 > > > > > > ------------------------------ > > Message: 3 > Date: Thu, 22 Dec 2011 13:11:05 -0600 > From: Martin Holste <mcholste(a)gmail.com> > Subject: Re: [syslog-ng] Pattern matching. > To: "Syslog-ng users' and developers' mailing list" > <syslog-ng(a)lists.balabit.hu> > Message-ID: > <CANpnLHgau7bZrSP2ro0QY=a8ZcJZLyqJgAVegWufDuszOjuCMA(a)mail.gmail.com > > > Content-Type: text/plain; charset=ISO-8859-1 > > You can also include an example pattern as part of the actual rule like > this: > > <ruleset> > <program></program> > <rule id="2"> > <pattern>@ESTRING:user::@ Security Microsoft > Windows security auditing.: [Success Audit] A computer account was > changed. Subject: Security ID: S-1-5-7 Account Name: > ANONYMOUS LOGON Account Domain: NT AUTHORITY Logon ID: 0x3e6 > Computer Account That Was Changed: Security ID: @ESTRING:: > @Account Name: @ESTRING:ACC_NAME: @ Account Domain: WW002 > Changed Attributes: SAM Account Name: - Display Name: - User > Principal Name: - Home Directory: - Home Drive: - Script Path: > - Profile Path: - User Workstations: - Password Last Set: > @ESTRING:: @@ESTRING:: @ Account Expires: - Primary Group ID: - > AllowedToDelegateTo: - Old UAC Value: - New UAC Value: - User > Account Control: - User Parameters: - SID History: - Logon > Hours: - DNS Host Name: - Service Principal Names: - > Additional Information: Privileges: - (EventID 4742)</pattern> > <examples> > <example> > <test_message > program="Microsoft_Windows_security_auditing.[5784]">: Security > Microsoft Windows security auditing.: [Success Audit] A computer > account was changed. Subject: Security ID: S-1-5-7 Account > Name: ANONYMOUS LOGON Account Domain: NT AUTHORITY Logon ID: > 0x3e6 Computer Account That Was Changed: Security ID: > S-1-5-21-776561741-789336058-725345543-305444 Account Name: User1$ > Account Domain: TEST Changed Attributes: SAM Account Name: - > Display Name: - User Principal Name: - Home Directory: - Home > Drive: - Script Path: - Profile Path: - User Workstations: - > Password Last Set: 12/22/2011 3:38:32 AM Account Expires: - > Primary Group ID: - AllowedToDelegateTo: - Old UAC Value: - New > UAC Value: - User Account Control: - User Parameters: - SID > History: - Logon Hours: - DNS Host Name: - Service Principal > Names: - Additional Information: Privileges: - (EventID > 4742)</test_message> > <test_value > name="ACC_NAME">User1$</test_value> > </example> > </examples> > </rule> > </ruleset> > > Then you can test it more easily like this: > pdbtool test patterndb.xml > > On Thu, Dec 22, 2011 at 8:04 AM, Balazs Scheidler <bazsi(a)balabit.hu> > wrote: > > On Thu, 2011-12-22 at 14:31 +0530, Anup Shetty wrote: > >> Nope, no luck yet. Still blanks being spit out. > >> > >> > >> Here's the exact extract of the pattern matching and the log: > >> > >> > >> Pattern String > >> --------------------------- > >> > >> > >> @ESTRING:user::@ Security Microsoft Windows security auditing.: > >> [Success Audit] A computer account was changed. ? ?Subject: ? Security > >> ID: ?S-1-5-7 ? Account Name: ?ANONYMOUS LOGON ? Account Domain: ?NT > >> AUTHORITY ? Logon ID: ?0x3e6 ? ?Computer Account That Was Changed: > >> Security ID: ?@ESTRING:: ?@Account Name: ? @ESTRING:ACC_NAME: @ > >> Account Domain: ?WW002 ? ?Changed Attributes: ? SAM Account Name: - > >> Display Name: ?- ? User Principal Name: - ? Home Directory: ?- ? Home > >> Drive: ?- ? Script Path: ?- ? Profile Path: ?- ? User Workstations: - > >> Password Last Set: @ESTRING:: @@ESTRING:: @ ? Account Expires: ?- > >> Primary Group ID: - ? AllowedToDelegateTo: - ? Old UAC Value: ?- ? New > >> UAC Value: ?- ? User Account Control: - ? User Parameters: - ? SID > >> History: ?- ? Logon Hours: ?- ? DNS Host Name: ?- ? Service Principal > >> Names: - ? ?Additional Information: ? Privileges: ?- (EventID 4742) > >> > >> > >> Log > >> ------------------ > >> > >> > >> Dec 22 03:38:32 Server.zoom11.test.net > >> Microsoft_Windows_security_auditing.[5784]: : Security Microsoft > >> Windows security auditing.: [Success Audit] A computer account was > >> changed. ? ?Subject: ? Security ID: ?S-1-5-7 ? Account Name: > >> ?ANONYMOUS LOGON ? Account Domain: ?NT AUTHORITY ? Logon ID: ?0x3e6 > >> ?Computer Account That Was Changed: ? Security ID: > >> ?S-1-5-21-776561741-789336058-725345543-305444 ? Account Name: ?User1$ > >> Account Domain: ?TEST ? ?Changed Attributes: ? SAM Account Name: - > >> Display Name: ?- ? User Principal Name: - ? Home Directory: ?- ? Home > >> Drive: ?- ? Script Path: ?- ? Profile Path: ?- ? User Workstations: - > >> Password Last Set: 12/22/2011 3:38:32 AM ? Account Expires: ?- > >> Primary Group ID: - ? AllowedToDelegateTo: - ? Old UAC Value: ?- ? New > >> UAC Value: ?- ? User Account Control: - ? User Parameters: - ? SID > >> History: ?- ? Logon Hours: ?- ? DNS Host Name: ?- ? Service Principal > >> Names: - ? ?Additional Information: ? Privileges: ?- (EventID 4742) > >> > >> > > "pdbtool match" can be used to test patterns. > > > > pdbtool patch -p <path to xml file> -P '<appname>' -M '<msg>' --debug > --color-out > > > > This even colours the output so that the partial matches can be > > recognized. This is the best way to troubleshoot patterns. > > > > -- > > Bazsi > > > > > -- Thanks and regards, Anup

2 1

How to split logs in the default folders?
by tokie＠tiscali.it 23 Dec '11

23 Dec '11

Hi, I'm not yet able to store my log by ip address. Can someone help me? How can I redirect everything that comes from a unique ip address (from LAN) to a folder I created? eg.: 192.168.1.1 (host1) == logs ==> 192.168.1.2 (syslog-ng-server )--->/ var/log/hosts/192.168.1.1 Tks and merry xmas!!! Tokie E' nata indoona: chiama, videochiama e messaggia Gratis. Scarica indoona per iPhone, Android e PC: http://www.indoona.com/

2 1

Solaris 10 support(?) in latest versions (e.g. 3.3.2)
by Marvin Nipper 22 Dec '11

22 Dec '11

Apologies upfront for the length, but this addresses two different problems. Environment: Solaris 10 x64 U10, fully patched (i.e. pretty much "new"). Currently operating (successfully) with 3.0.5, compiled on that platform, and distributed to a number of other Solaris 10 x64 targets. Problem: Trying to "make the leap" to 3.3.2, and get current, but that version seems to have numerous Solaris compatibility issues. Just trying to understand the status of Solaris support. Problem #1 Latest packaging no longer works with Solaris-based "make". After a clean configure (output here): syslog-ng Open Source Edition 3.3.2 configured Compiler options: compiler : gcc -std=gnu99 compiler options : -g -O2 -Wall -pthread -D_REENTRANT -D_PTHREADS -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -I/usr/local/include/eventlog -I/usr/local/ssl/include -DLIBNET_LIL_ENDIAN -I$(top_srcdir)/lib/ivykis/lib/include -I$(top_builddir)/lib/ivykis/lib/include -I$(top_srcdir)/lib/ivykis/modules/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 linker flags : -R/usr/local/lib -lpthread prefix : /usr/local linking mode : dynamic __thread keyword : yes Submodules: ivykis : internal libmongo-client : internal Features: Debug symbols : no GCC profiling : no Memtrace : no IPV6 support : yes spoof-source support : yes tcp-wrapper support : no Linux capability support : no PCRE support : no Env wrapper support : no systemd support : no (unit dir: none) Modules: Module search path : /usr/local/lib/syslog-ng Default module list : affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat,afstreams Sun STREAMS support (module): yes SSL support (module) : yes SQL support (module) : no PACCT module (EXPERIMENTAL) : no MongoDB destination (module): yes JSON support (module) : no (using no) ..... a simple "make" attempt now immediately results in this: # make make all-recursive Making all in lib make: Fatal error in reader: Makefile, line 991: Unexpected end of line seen Current working directory /var/opt/packages/syslog-ng-3.3.2/lib *** Error code 1 The following command caused the error: fail= failcom='exit 1'; \ for f in x $MAKEFLAGS; do \ case $f in \ *=* | --[!k]*);; \ *k*) failcom='fail=yes';; \ esac; \ done; \ dot_seen=no; \ target=`echo all-recursive | sed s/-recursive//`; \ list='lib modules syslog-ng scripts tests doc contrib scl debian tgz2build build'; for subdir in $list; do \ echo "Making $target in $subdir"; \ if test "$subdir" = "."; then \ dot_seen=yes; \ local_target="$target-am"; \ else \ local_target="$target"; \ fi; \ (CDPATH="${ZSH_VERSION+.}:" && cd $subdir && make $local_target) \ || eval $failcom; \ done; \ if test "$dot_seen" = "no"; then \ make "$target-am" || exit 1; \ fi; test -z "$fail" make: Fatal error: Command failed for target `all-recursive' Current working directory /var/opt/packages/syslog-ng-3.3.2 *** Error code 1 make: Fatal error: Command failed for target `all' Not being a developer, per se, I spent time in Google-land, the general result that I find is: "These are caused because Solaris "make" doesn't accept the 'GNU multi-line Makefile format'. Use gmake instead." But there are obviously folks who also say that this can be fixed by "correcting" the problems in the Makefiles (so that would seem to be a preferable alternative). Problem #2 Can no longer perform successful compilations, due to this error: ld: fatal: relocation error: R_386_GOTOFF Same configure output as previously shown, followed by gmake (to get around the first problem). Numerous modules compile successfully, but then this happens: gcc -DHAVE_CONFIG_H -I. -I../.. -D_GNU_SOURCE -I../../lib/include -I../../lib/include -I../../modules/include -Wall -g -O2 -D_REENTRANT -D_POSIX_C_SOURCE=199506L -D_POSIX_PTHREAD_SEMANTICS -D_XPG4_2 -MT iv_event_test.o -MD -MP -MF .deps/iv_event_test.Tpo -c -o iv_event_test.o iv_event_test.c mv -f .deps/iv_event_test.Tpo .deps/iv_event_test.Po /bin/bash ../../libtool --tag=CC --mode=link gcc -Wall -g -O2 -D_REENTRANT -D_POSIX_C_SOURCE=199506L -D_POSIX_PTHREAD_SEMANTICS -D_XPG4_2 -R/usr/local/lib -o iv_event_test iv_event_test.o ../../lib/libivykis.la ../libivykis-modules.la -lrt -lpthread -lsocket libtool: link: gcc -Wall -g -O2 -D_REENTRANT -D_POSIX_C_SOURCE=199506L -D_POSIX_PTHREAD_SEMANTICS -D_XPG4_2 -o iv_event_test iv_event_test.o ../../lib/.libs/libivykis.a ../.libs/libivykis-modules.a /var/opt/packages/syslog-ng-3.3.2/lib/ivykis/lib/.libs/libivykis.a -lrt -lpthread -lsocket -R/usr/local/lib ld: fatal: relocation error: R_386_GOTOFF: file ../.libs/libivykis-modules.a(iv_event.o): symbol __tls: relocation illegal for TLS symbol ld: fatal: relocation error: R_386_GOTOFF: file ../.libs/libivykis-modules.a(iv_event.o): symbol __tls: relocation illegal for TLS symbol ld: fatal: relocation error: R_386_GOTOFF: file ../.libs/libivykis-modules.a(iv_event.o): symbol __tls: relocation illegal for TLS symbol ld: fatal: relocation error: R_386_GOTOFF: file ../.libs/libivykis-modules.a(iv_event.o): symbol __tls: relocation illegal for TLS symbol collect2: ld returned 1 exit status gmake[7]: *** [iv_event_test] Error 1 gmake[7]: Leaving directory `/var/opt/packages/syslog-ng-3.3.2/lib/ivykis/modules/test' gmake[6]: *** [all-recursive] Error 1 gmake[6]: Leaving directory `/var/opt/packages/syslog-ng-3.3.2/lib/ivykis/modules' gmake[5]: *** [all-recursive] Error 1 gmake[5]: Leaving directory `/var/opt/packages/syslog-ng-3.3.2/lib/ivykis' gmake[4]: *** [all] Error 2 gmake[4]: Leaving directory `/var/opt/packages/syslog-ng-3.3.2/lib/ivykis' gmake[3]: *** [all-recursive] Error 1 gmake[3]: Leaving directory `/var/opt/packages/syslog-ng-3.3.2/lib' gmake[2]: *** [all] Error 2 gmake[2]: Leaving directory `/var/opt/packages/syslog-ng-3.3.2/lib' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/var/opt/packages/syslog-ng-3.3.2' gmake: *** [all] Error 2 Again, adopting the "Google is my friend" approach, this appears to be an issue with "Solaris not supporting Visibility". (http://www.ilkda.com/compile/Build_Problems.htm) Most of the sites where I see this being discussed, with other (various) source components, I also see the developers subsequently modifying their environment to "adjust for" the Visibility issue in Solaris. So, again, not being a C developer myself, I guess I'm wondering whether I'm just doing something wrong with 3.3.2 (i.e. something that I need to do different from my 3.0.5 compilation), or whether this is truly just a "Solaris thing" that has not been accounted for in the current code base? Sorry for not being better at sorting this out myself. Any and all help/guidance would certainly be appreciated. GLAD to do some sort of testing here, if that will help. Also, just for reference, here is the full "configure" output from the 3.0.5 build, that compiles clean, and works just fine: checking pkg-config is at least version 0.9.0... yes checking for a BSD-compatible install... ./install-sh -c checking whether build environment is sane... yes checking for a thread-safe mkdir -p... ./install-sh -c -d checking for gawk... no checking for mawk... no checking for nawk... nawk checking whether make sets $(MAKE)... yes checking for gcc... gcc checking for C compiler default output file name... a.out checking whether the C compiler works... yes checking whether we are cross compiling... no checking for suffix of executables... checking for suffix of object files... o checking whether we are using the GNU C compiler... yes checking whether gcc accepts -g... yes checking for gcc option to accept ISO C89... none needed checking for style of include used by make... GNU checking dependency style of gcc... gcc3 checking for gcc... (cached) gcc checking whether we are using the GNU C compiler... (cached) yes checking whether gcc accepts -g... (cached) yes checking for gcc option to accept ISO C89... (cached) none needed checking dependency style of gcc... (cached) gcc3 checking for bison... bison -y checking for flex... flex checking lex output file root... lex.yy checking lex library... -lfl checking whether yytext is a pointer... yes checking whether make sets $(MAKE)... (cached) yes checking for ranlib... ranlib checking for special C compiler options needed for large files... no checking for _FILE_OFFSET_BITS value needed for large files... 64 checking how to enable static linking for certain libraries... GNU or Solaris checking how to run the C preprocessor... gcc -E checking for grep that handles long lines and -e... /usr/sfw/bin/ggrep checking for egrep... /usr/sfw/bin/ggrep -E checking for ANSI C header files... yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes checking for memory.h... yes checking for strings.h... yes checking for inttypes.h... yes checking for stdint.h... yes checking for unistd.h... yes checking dmalloc.h usability... no checking dmalloc.h presence... no checking for dmalloc.h... no checking for strings.h... (cached) yes checking getopt.h usability... yes checking getopt.h presence... yes checking for getopt.h... yes checking stropts.h usability... yes checking stropts.h presence... yes checking for stropts.h... yes checking sys/strlog.h usability... yes checking sys/strlog.h presence... yes checking for sys/strlog.h... yes checking door.h usability... yes checking door.h presence... yes checking for door.h... yes checking sys/capability.h usability... no checking sys/capability.h presence... no checking for sys/capability.h... no checking sys/prctl.h usability... no checking sys/prctl.h presence... no checking for sys/prctl.h... no checking tcpd.h usability... yes checking tcpd.h presence... yes checking for tcpd.h... yes checking whether struct tm is in sys/time.h or time.h... time.h checking for struct tm.tm_gmtoff... no checking for I_CONSLOG... yes checking for O_LARGEFILE... yes checking for struct sockaddr_storage... yes checking for struct sockaddr_in6... yes checking for PR_SET_KEEPCAPS... no checking for door_create in -ldoor... yes checking for socket in -lsocket... yes checking for nanosleep in -lrt... yes checking for gethostbyname in -lnsl... yes checking for regexec in -lregex... no checking for res_init in -lresolv... yes checking for cap_set_proc in -lcap... no checking for PCRE... no configure: WARNING: Cannot find pcre version >= 7.3. pcre support disabled. checking for strdup... yes checking for strtol... yes checking for strtoll... yes checking for strtoimax... yes checking for inet_aton... no checking for inet_ntoa... no checking for getopt_long... yes checking for getaddrinfo... no checking for getutent... yes checking for pread... yes checking for pwrite... yes checking for strcasestr... no checking for TCP wrapper library... -lwrap checking for OPENSSL... yes checking for dlsym in -ldl... yes checking for inflate in -lz... yes checking for dlsym in -ldl... (cached) yes checking for LIBDBI... no checking for dbi_initialize in -ldbi... no checking pthread.h usability... yes checking pthread.h presence... yes checking for pthread.h... yes checking for pthread_create in -lpthread... yes checking for LIBNET... stty: : No such device or address stty: : No such device or address yes checking whether to enable Sun STREAMS support... yes checking whether to enable Sun door support... yes checking whether to enable IPv6 support... yes checking whether to enable SQL support... no checking whether to enable Linux capability support... no checking whether to enable PCRE support... no checking pkg-config is at least version 0.14... yes checking for GLIB - version >= 2.10.1... yes (version 2.13.5) checking for EVTLOG... yes checking for LIBNET... stty: : No such device or address stty: : No such device or address yes checking for static GLib libraries... yes checking sanity checking Glib headers... yes configure: creating ./config.status config.status: creating dist.conf config.status: creating Makefile config.status: creating syslog-ng.spec config.status: creating src/Makefile config.status: creating doc/Makefile config.status: creating doc/docvars.xml config.status: creating contrib/Makefile config.status: creating tests/Makefile config.status: creating debian/Makefile config.status: creating tgz2build/Makefile config.status: creating tests/unit/Makefile config.status: creating tests/functional/Makefile config.status: creating tests/loggen/Makefile config.status: creating config.h config.status: executing depfiles commands syslog-ng Open Source Edition 3.0.5 configured Compiler options: compiler : gcc compiler options : -g -O2 -Wall -I/usr/local/include/glib-2.0 -I/u sr/local/lib/glib-2.0/include -I/usr/local/include/eventlog -I/usr/local/ssl /include -DLIBNET_LIL_ENDIAN -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET _BITS=64 linker flags : -R/usr/local/lib -lpthread prefix : /usr/local linking mode : dynamic Features: Sun STREAMS support : yes Sun Door support : yes Debug symbols : no GCC profiling : no Memtrace : no IPV6 support : yes spoof-source support : yes tcp-wrapper support : no SSL support : yes SQL support : no Linux capability support : no PCRE support : no Env wrapper support : no This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden.

4 15

Pattern matching.
by Anup Shetty 22 Dec '11

22 Dec '11

I am trying to match the pattern for DC logs and here is my XML format Here's the patterndb.xml file at /opt/syslog-ng/var/patterndb.xml" --------------------------------------- <?xml version='1.0' encoding='UTF-8'?> <patterndb version='3' pub_date='2011-12-21'> <ruleset id='90c9b341f4e3d63c5ed8b29950491bf8' name='Domain Ctrls'> <rules> <rule provider='localtest' id='012c230f236d6a3f761ba956e7dff26a' class='system'> <patterns> <pattern> @ESTRING:user::@ Security Microsoft Windows security auditing.: [Success Audit] A computer account was changed. Subject: Security ID: S-1-5-7 Account Name: ANONYMOUS LOGON Account Domain: NT AUTHORITY Logon ID: 0x3e6 Computer Account That Was Changed: Security ID: @ESTRING:: @Account Name: @ESTRING:*ACC_NAME*: @ Account Domain: testdomain Changed Attributes: SAM Account Name: - Display Name: - User Principal Name: - Home Directory: - Home Drive: - Script Path: - Profile Path: - User Workstations: - Password Last Set: @ESTRING:: @@ESTRING:: @ Account Expires: - Primary Group ID: - AllowedToDelegateTo: - Old UAC Value: - New UAC Value: - User Account Control: - User Parameters: - SID History: - Logon Hours: - DNS Host Name: - Service Principal Names: - Additional Information: Privileges: - (EventID 4742) </pattern> </patterns> </rule> </rules> </ruleset> </patterndb> --------------------------------------- Here's the syslog-ng conf extract: --------------------------------------- parser pattern_db { db_parser( file("/opt/syslog-ng/var/patterndb.xml") ); }; destination patt_d{ file("/data/test/${R_YEAR}/${R_MONTH}/${R_DAY}/Domain_Ctrl__${SOURCEIP}_${R_YEAR}_${R_MONTH}_${R_DAY}.log" owner("test") group("test") perm(0660) dir-owner("test") dir-group("test") dir-perm(0770) template("$*ACC_NAME*\n $MSG\n") ); }; --------------------------------- but the *ACC_NAME* returns blank, although the log contains that field. -- Thanks Anup

4 4

Quick filter question again
by Lay, James 22 Dec '11

22 Dec '11

Hey all! Hope this is quick/easy. So I'm matching IP's that I don't want to see...filter below: message("66\.220\.") My issues is, this will match not only say 66.220.1.1, but also 166.220.1.1. I've tried changing to: message(" 66\.220\.") but that'd didn't fly. Any suggestions on how to more exactly match IP blocks? Thanks all. James

2 2

[PATCH (3.4) 0/1]: value-pairs: Add value_pairs_foreach_sorted().
by Gergely Nagy 22 Dec '11

22 Dec '11

The patch that will follow shortly implements a value_pairs_foreach_sorted() function, that can be used for multiple purposes in the future: - Template functions that want to format their output with keys always in a predictable order. - Various other parts of the code, that work better when operating on a sorted collection. I'm submitting it separately, so that it'll be easier for me to build upon it in the not too distant future. (And because I almost forgot to send it, even though it was origianlly made in early november...)

2 3