December 2012 - syslog-ng - lists.balabit.hu

FreeBSD 3.4 pre beta testing
by Peter Czanik 14 Dec '12

14 Dec '12

Hello, Attached is a diff against the current sysutils/syslog-nd-devel port. Please test it, so problems could be discovered before the beta release. To use the attached diff, change to the /usr/ports/sysutils/syslog-ng-devel directory and run: patch -p1 < /path/to/prebeta.diff Changes since alpha3: - latest sources - redundant patches removed - glib json support removed (json parser needs json-c) - added patch to fix libnet (will be part of the beta release) - lot of pkg-list changes Once 3.4 beta is released I plan to update the port too. Happy testing! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

3.4 head on FreeBSD
by Peter Czanik 12 Dec '12

12 Dec '12

Hello, While trying to compile on FreeBSD 9.1, I ran into this problem: /bin/sh ../../libtool --tag=CC --mode=compile cc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -D_REENTRANT -I/usr/local/include/glib-2.0 -I/usr/local/include/eventlog -I/usr/local/include -I/usr/local/include -I../../lib/ivykis/lib/include -I../../lib/ivykis/lib/include -I../../lib/ivykis/modules/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -pipe -I/usr/local/include/libnet116 -I/usr/local/include -fno-strict-aliasing -Wall -g -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c -o libafsocket_notls_la-afinet-dest.lo `test -f 'afinet-dest.c' || echo './'`afinet-dest.c libtool: compile: cc -std=gnu99 -DHAVE_CONFIG_H -I. -I../.. -I../../lib -I../../lib -D_REENTRANT -I/usr/local/include/glib-2.0 -I/usr/local/include/eventlog -I/usr/local/include -I/usr/local/include -I../../lib/ivykis/lib/include -I../../lib/ivykis/lib/include -I../../lib/ivykis/modules/include -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -O2 -pipe -I/usr/local/include/libnet116 -I/usr/local/include -fno-strict-aliasing -Wall -g -pthread -MT libafsocket_notls_la-afinet-dest.lo -MD -MP -MF .deps/libafsocket_notls_la-afinet-dest.Tpo -c afinet-dest.c -fPIC -DPIC -o .libs/libafsocket_notls_la-afinet-dest.o In file included from /usr/local/include/libnet116/libnet.h:120, from afinet-dest.c:43: /usr/local/include/libnet116/./libnet/libnet-structures.h:235: error: redefinition of typedef 'libnet_t' afinet-dest.h:35: error: previous declaration of 'libnet_t' was here gmake[4]: *** [libafsocket_notls_la-afinet-dest.lo] Error 1 gmake[4]: Leaving directory `/root/syslog-ng-devel/work/syslog-ng-3.4.0alpha3/modules/afsocket' gmake[3]: *** [all] Error 2 gmake[3]: Leaving directory `/root/syslog-ng-devel/work/syslog-ng-3.4.0alpha3/modules/afsocket' gmake[2]: *** [all-recursive] Error 1 gmake[2]: Leaving directory `/root/syslog-ng-devel/work/syslog-ng-3.4.0alpha3/modules' gmake[1]: *** [all-recursive] Error 1 gmake[1]: Leaving directory `/root/syslog-ng-devel/work/syslog-ng-3.4.0alpha3' gmake: *** [all] Error 2 *** [do-build] Error code 1 Disabling libnet support results in a successful build, but it can't be started: root@fb91:/root/syslog-ng-devel # /usr/local/etc/rc.d/syslog-ng onerestart syslog_ng not running? (check /var/run/syslog.pid). Starting syslog_ng. WARNING: Configuration file format is too old, syslog-ng is running in compatibility mode Please update it to use the syslog-ng 3.4 format at your time of convinience, compatibility mode can operate less efficiently in some cases. To upgrade the configuration, please review the warnings about incompatible changes printed by syslog-ng, and once completed change the @version header at the top of the configuration file.; Unable to determine how to monitor this fd, follow_freq() not set and it is not possible to poll it with the current ivykis polling method, try changing IV_EXCLUDE_POLL_METHOD environment variable; fd='11' Error initializing log_reader, closing fd; fd='11' Error initializing message pipeline; /usr/local/etc/rc.d/syslog-ng: WARNING: failed to start syslog_ng FreeBSD installs syslog-ng-3.4.0alpha3/scl/syslog-ng.conf by default, so it needs a version bump. It can easily be fixed by me. But system() is now in C instead of shell scripts... Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

2 3

WARNING: window sizing for tcp sources
by Anton Koldaev 11 Dec '12

11 Dec '12

Hello, I have two questions: 1. % sudo /etc/init.d/syslog-ng restart * Stopping system logging syslog-ng [ OK ] * Starting system logging syslog-ng WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='80', new_log_iw_size='100', min_log_fifo_size='25000' I have a lot of tcp sources, destination and options. Where should I look for this warning? I don't have "log_iw_size=80" anywhere. 2. `syslog-ng --preprocess-into=/tmp/pre.conf` ^Does it work for anybody? It writes the config file but it hangs after it for me. Is it supposed to be used when syslog-ng daemon is down? -- Best regards, Koldaev Anton

2 4

syslog-ng 3.4beta
by Balazs Scheidler 10 Dec '12

10 Dec '12

Hi, Having finally been able to finish the network() source and destination drivers I declare 3.4 as feature complete, and plan to publish 3.4beta1 fairly soon. Let's say, before the winter holiday. In hindsight, this took much longer than it should have, 3.4 has grown a lot of interesting functionality: - junctions and more complicated log processing paths - amqp destination - smtp destination - json output and parser plugins - key rewriting in destination drivers (add, remove prefixes) - a number of new template functions (uuid, hash functions, etc) - a number of new parsers in db-parser (set, pcre, ...) - allow 32 bit message and nvpair sizes - improved systemd support - demand loaded plugins - plugin support for log transport protocols and the network driver to be able to use them easily And other things I've probably forgot. I have just merged Algernon's accumulated merge queue, and will probably cherry-pick stuff before the final release and merge in the fixes from 3.3 before the beta release. If you have stuff that you want to see in the beta, please speak up now. Testing _before_ the beta would also be appreciated, so that the beta is actually usable by laymen. Thanks :) As always, the latest code is in git, Algernon builds daily snapshots. Thanks -- Bazsi

5 9

[PATCH 1/2] pdbtool: match with supplied message did not perform processing
by balint.kovacs＠balabit.com 10 Dec '12

10 Dec '12

From: Balint Kovacs <blint(a)blint.hu> Under some circumstances pdbtool match did not process the message if it was provided on the command line due to a regression introduced as part of a LogProto refactoring (2d0fb042) Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/pdbtool/pdbtool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/dbparser/pdbtool/pdbtool.c b/modules/dbparser/pdbtool/pdbtool.c index deab84d..25014fe 100644 --- a/modules/dbparser/pdbtool/pdbtool.c +++ b/modules/dbparser/pdbtool/pdbtool.c @@ -503,7 +503,7 @@ pdbtool_match(int argc, char *argv[]) { dbg_list = g_array_new(FALSE, FALSE, sizeof(RDebugInfo)); } - while (!eof && buf) + while (!eof && (buf || match_message)) { invalidate_cached_time(); if (G_LIKELY(proto)) -- 1.7.9.5

1 1

[PATCH] afsocket: Add an ip-protocol() option for the network driver
by Gergely Nagy 08 Dec '12

08 Dec '12

To make it easy to set up IPv6 sources or destinations, the network() driver should have an option to select the IP protocol. This patch does just that, in the form of an ip-protocol() option, which allows one to select either IPv4 or IPv6. In the end, this allows us to have a source as follows: source s_v6_syslog { network(port(12345) transport(framed) ip-protocol(6)); }; The option is available for both the source and destination version of network(). Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afsocket-grammar.ym | 29 ++++++++++++++++++++++++++++- modules/afsocket/afsocket-parser.c | 1 + 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/modules/afsocket/afsocket-grammar.ym b/modules/afsocket/afsocket-grammar.ym index 7beb6e8..464d5d8 100644 --- a/modules/afsocket/afsocket-grammar.ym +++ b/modules/afsocket/afsocket-grammar.ym @@ -79,6 +79,7 @@ TLSContext *last_tls_context; %token KW_UDP6 %token KW_NETWORK %token KW_TRANSPORT +%token KW_IP_PROTOCOL %token KW_IP_TTL %token KW_SO_BROADCAST @@ -136,6 +137,8 @@ TLSContext *last_tls_context; %type <ptr> dest_afnetwork %type <ptr> dest_afnetwork_params +%type <num> afinet_ip_protocol_option + %% start @@ -149,6 +152,25 @@ start | LL_CONTEXT_DESTINATION dest_afnetwork { YYACCEPT; } ; +afinet_ip_protocol_option + : LL_NUMBER + { + CHECK_ERROR($1 == 4 || $1 == 6, @1, "ip-protocol option can only be 4 or 6!"); + if ($1 == 4) + { + $$ = AF_INET; + } + else + { + $$ = AF_INET6; + } + } + ; + +source_afinet_ip_protocol + : KW_IP_PROTOCOL '(' afinet_ip_protocol_option ')' { ((AFSocketSourceDriver *)last_driver)->address_family = $3; } + ; + source_afunix : KW_UNIX_DGRAM '(' source_afunix_dgram_params ')' { $$ = $3; } | KW_UNIX_STREAM '(' source_afunix_stream_params ')' { $$ = $3; } @@ -314,6 +336,7 @@ source_afnetwork_option : source_afinet_option | source_afsocket_transport | source_afsocket_stream_params {} + | source_afinet_ip_protocol ; source_afsocket_transport @@ -335,7 +358,10 @@ source_afsocket_transport } ; - +dest_afinet_ip_protocol + : KW_IP_PROTOCOL '(' afinet_ip_protocol_option ')' { ((AFSocketDestDriver *)last_driver)->address_family = $3; } + ; + dest_afunix : KW_UNIX_DGRAM '(' dest_afunix_dgram_params ')' { $$ = $3; } | KW_UNIX_STREAM '(' dest_afunix_stream_params ')' { $$ = $3; } @@ -500,6 +526,7 @@ dest_afnetwork_options dest_afnetwork_option : dest_afinet_option | dest_afsocket_transport + | dest_afinet_ip_protocol ; dest_afsocket_transport diff --git a/modules/afsocket/afsocket-parser.c b/modules/afsocket/afsocket-parser.c index 4065d31..f9e14a4 100644 --- a/modules/afsocket/afsocket-parser.c +++ b/modules/afsocket/afsocket-parser.c @@ -72,6 +72,7 @@ static CfgLexerKeyword afsocket_keywords[] = { { "tcp_keepalive_intvl", KW_TCP_KEEPALIVE_INTVL, 0x0304 }, { "spoof_source", KW_SPOOF_SOURCE }, { "transport", KW_TRANSPORT }, + { "ip_protocol", KW_IP_PROTOCOL }, { "max_connections", KW_MAX_CONNECTIONS }, { "keep_alive", KW_KEEP_ALIVE }, { NULL } -- 1.7.10.4

2 1

syslog-ng OSE 3.4 Administrator Guide draft update
by Fekete Róbert 07 Dec '12

07 Dec '12

Hi, My colleague, Ancsur Sebestyen had some time this week to work on the 3.4 admin guide a bit. Still far from being finished or up-to-date, the following new stuff has been documented since the first draft: * The tfhash template function * The JSON parser * The --disable-json compile option * The @PCRE@ patterndb parser * Bug 204: dbparser docs not clear for @NUMBER@ * Bug 205 Docs for macro MSGHDR incorrect * The LOGHOST macro * The syslog-ng-ctl reload option is included in the syslog-ng-ctl man page Also, the links to the docs have changed. The new ones are: HTML: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… Single-page HTML: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… PDF: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… As always, any feedback is welcome. Robert Fekete

2 1

[PATCH] system(): Use follow-freq(1) on FreeBSD prior to 9.1
by Gergely Nagy 07 Dec '12

07 Dec '12

FreeBSD prior to 9.1 does not have a kqueue-pollable /dev/klog, so syslog-ng would fall back to using timers, except that follow-freq() is explicitly set to zero, also diabling timer-based polling. This has the effect of making syslog-ng not read from /dev/klog at all. To work around the issue, on FreeBSD prior to 9.1, use follow-freq(1) for /dev/klog when using the system() source. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- scl/system/generate-system-source.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/scl/system/generate-system-source.sh b/scl/system/generate-system-source.sh index 708a7a4..34cc009 100755 --- a/scl/system/generate-system-source.sh +++ b/scl/system/generate-system-source.sh @@ -74,7 +74,17 @@ EOF cat <<EOF unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); -file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse)); +EOF + case "$osversion" in + 7.*|8.*|9.0*) + ffreq=1 + ;; + *) + ffreq=0 + ;; + esac + cat <<EOF +file("/dev/klog" follow-freq($ffreq) program-override("kernel") flags(no-parse)); EOF ;; GNU/kFreeBSD) -- 1.7.10.4

2 1

Embedded log path gives syntax error
by Anton Koldaev 06 Dec '12

06 Dec '12

https://gist.github.com/40c233f40ebc8bc9163e -- Best regards, Koldaev Anton

2 1

openSUSE packages updated, SLES 11 SP2 added
by Peter Czanik 06 Dec '12

06 Dec '12

Hello, My openSUSE packages are now updated. There were two major changes: - binaries moved from / to /usr (supported by openSUSE 12.1+ and SLES 11 SP2) - lot of legacy ifdefs removed, so old releases ( <12.1, unsupported by the openSUSE project) are not supported any more This has two advantages: a lot cleaner spec file, and SLES support added. SLES now also has SSL support. The SLES package was tested only on SLES 11 SP2, but will most likely also work on older SLES 11 releases as long as /usr is not a separate partition. As usual, my 3.3 packages are available at https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 Once 3.4 beta is released, I'll rework my 3.4 packages too. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0