- syslog-ng - lists.balabit.hu

WARNING: window sizing for tcp sources
by Anton Koldaev 11 Dec '12

11 Dec '12

Hello, I have two questions: 1. % sudo /etc/init.d/syslog-ng restart * Stopping system logging syslog-ng [ OK ] * Starting system logging syslog-ng WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='80', new_log_iw_size='100', min_log_fifo_size='25000' I have a lot of tcp sources, destination and options. Where should I look for this warning? I don't have "log_iw_size=80" anywhere. 2. `syslog-ng --preprocess-into=/tmp/pre.conf` ^Does it work for anybody? It writes the config file but it hangs after it for me. Is it supposed to be used when syslog-ng daemon is down? -- Best regards, Koldaev Anton

2 4

syslog-ng 3.4beta
by Balazs Scheidler 10 Dec '12

10 Dec '12

Hi, Having finally been able to finish the network() source and destination drivers I declare 3.4 as feature complete, and plan to publish 3.4beta1 fairly soon. Let's say, before the winter holiday. In hindsight, this took much longer than it should have, 3.4 has grown a lot of interesting functionality: - junctions and more complicated log processing paths - amqp destination - smtp destination - json output and parser plugins - key rewriting in destination drivers (add, remove prefixes) - a number of new template functions (uuid, hash functions, etc) - a number of new parsers in db-parser (set, pcre, ...) - allow 32 bit message and nvpair sizes - improved systemd support - demand loaded plugins - plugin support for log transport protocols and the network driver to be able to use them easily And other things I've probably forgot. I have just merged Algernon's accumulated merge queue, and will probably cherry-pick stuff before the final release and merge in the fixes from 3.3 before the beta release. If you have stuff that you want to see in the beta, please speak up now. Testing _before_ the beta would also be appreciated, so that the beta is actually usable by laymen. Thanks :) As always, the latest code is in git, Algernon builds daily snapshots. Thanks -- Bazsi

5 9

[PATCH 1/2] pdbtool: match with supplied message did not perform processing
by balint.kovacs＠balabit.com 10 Dec '12

10 Dec '12

From: Balint Kovacs <blint(a)blint.hu> Under some circumstances pdbtool match did not process the message if it was provided on the command line due to a regression introduced as part of a LogProto refactoring (2d0fb042) Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/pdbtool/pdbtool.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/dbparser/pdbtool/pdbtool.c b/modules/dbparser/pdbtool/pdbtool.c index deab84d..25014fe 100644 --- a/modules/dbparser/pdbtool/pdbtool.c +++ b/modules/dbparser/pdbtool/pdbtool.c @@ -503,7 +503,7 @@ pdbtool_match(int argc, char *argv[]) { dbg_list = g_array_new(FALSE, FALSE, sizeof(RDebugInfo)); } - while (!eof && buf) + while (!eof && (buf || match_message)) { invalidate_cached_time(); if (G_LIKELY(proto)) -- 1.7.9.5

1 1

[PATCH] afsocket: Add an ip-protocol() option for the network driver
by Gergely Nagy 08 Dec '12

08 Dec '12

To make it easy to set up IPv6 sources or destinations, the network() driver should have an option to select the IP protocol. This patch does just that, in the form of an ip-protocol() option, which allows one to select either IPv4 or IPv6. In the end, this allows us to have a source as follows: source s_v6_syslog { network(port(12345) transport(framed) ip-protocol(6)); }; The option is available for both the source and destination version of network(). Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsocket/afsocket-grammar.ym | 29 ++++++++++++++++++++++++++++- modules/afsocket/afsocket-parser.c | 1 + 2 files changed, 29 insertions(+), 1 deletion(-) diff --git a/modules/afsocket/afsocket-grammar.ym b/modules/afsocket/afsocket-grammar.ym index 7beb6e8..464d5d8 100644 --- a/modules/afsocket/afsocket-grammar.ym +++ b/modules/afsocket/afsocket-grammar.ym @@ -79,6 +79,7 @@ TLSContext *last_tls_context; %token KW_UDP6 %token KW_NETWORK %token KW_TRANSPORT +%token KW_IP_PROTOCOL %token KW_IP_TTL %token KW_SO_BROADCAST @@ -136,6 +137,8 @@ TLSContext *last_tls_context; %type <ptr> dest_afnetwork %type <ptr> dest_afnetwork_params +%type <num> afinet_ip_protocol_option + %% start @@ -149,6 +152,25 @@ start | LL_CONTEXT_DESTINATION dest_afnetwork { YYACCEPT; } ; +afinet_ip_protocol_option + : LL_NUMBER + { + CHECK_ERROR($1 == 4 || $1 == 6, @1, "ip-protocol option can only be 4 or 6!"); + if ($1 == 4) + { + $$ = AF_INET; + } + else + { + $$ = AF_INET6; + } + } + ; + +source_afinet_ip_protocol + : KW_IP_PROTOCOL '(' afinet_ip_protocol_option ')' { ((AFSocketSourceDriver *)last_driver)->address_family = $3; } + ; + source_afunix : KW_UNIX_DGRAM '(' source_afunix_dgram_params ')' { $$ = $3; } | KW_UNIX_STREAM '(' source_afunix_stream_params ')' { $$ = $3; } @@ -314,6 +336,7 @@ source_afnetwork_option : source_afinet_option | source_afsocket_transport | source_afsocket_stream_params {} + | source_afinet_ip_protocol ; source_afsocket_transport @@ -335,7 +358,10 @@ source_afsocket_transport } ; - +dest_afinet_ip_protocol + : KW_IP_PROTOCOL '(' afinet_ip_protocol_option ')' { ((AFSocketDestDriver *)last_driver)->address_family = $3; } + ; + dest_afunix : KW_UNIX_DGRAM '(' dest_afunix_dgram_params ')' { $$ = $3; } | KW_UNIX_STREAM '(' dest_afunix_stream_params ')' { $$ = $3; } @@ -500,6 +526,7 @@ dest_afnetwork_options dest_afnetwork_option : dest_afinet_option | dest_afsocket_transport + | dest_afinet_ip_protocol ; dest_afsocket_transport diff --git a/modules/afsocket/afsocket-parser.c b/modules/afsocket/afsocket-parser.c index 4065d31..f9e14a4 100644 --- a/modules/afsocket/afsocket-parser.c +++ b/modules/afsocket/afsocket-parser.c @@ -72,6 +72,7 @@ static CfgLexerKeyword afsocket_keywords[] = { { "tcp_keepalive_intvl", KW_TCP_KEEPALIVE_INTVL, 0x0304 }, { "spoof_source", KW_SPOOF_SOURCE }, { "transport", KW_TRANSPORT }, + { "ip_protocol", KW_IP_PROTOCOL }, { "max_connections", KW_MAX_CONNECTIONS }, { "keep_alive", KW_KEEP_ALIVE }, { NULL } -- 1.7.10.4

2 1

syslog-ng OSE 3.4 Administrator Guide draft update
by Fekete Róbert 07 Dec '12

07 Dec '12

Hi, My colleague, Ancsur Sebestyen had some time this week to work on the 3.4 admin guide a bit. Still far from being finished or up-to-date, the following new stuff has been documented since the first draft: * The tfhash template function * The JSON parser * The --disable-json compile option * The @PCRE@ patterndb parser * Bug 204: dbparser docs not clear for @NUMBER@ * Bug 205 Docs for macro MSGHDR incorrect * The LOGHOST macro * The syslog-ng-ctl reload option is included in the syslog-ng-ctl man page Also, the links to the docs have changed. The new ones are: HTML: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… Single-page HTML: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… PDF: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… As always, any feedback is welcome. Robert Fekete

2 1

[PATCH] system(): Use follow-freq(1) on FreeBSD prior to 9.1
by Gergely Nagy 07 Dec '12

07 Dec '12

FreeBSD prior to 9.1 does not have a kqueue-pollable /dev/klog, so syslog-ng would fall back to using timers, except that follow-freq() is explicitly set to zero, also diabling timer-based polling. This has the effect of making syslog-ng not read from /dev/klog at all. To work around the issue, on FreeBSD prior to 9.1, use follow-freq(1) for /dev/klog when using the system() source. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- scl/system/generate-system-source.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/scl/system/generate-system-source.sh b/scl/system/generate-system-source.sh index 708a7a4..34cc009 100755 --- a/scl/system/generate-system-source.sh +++ b/scl/system/generate-system-source.sh @@ -74,7 +74,17 @@ EOF cat <<EOF unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); -file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse)); +EOF + case "$osversion" in + 7.*|8.*|9.0*) + ffreq=1 + ;; + *) + ffreq=0 + ;; + esac + cat <<EOF +file("/dev/klog" follow-freq($ffreq) program-override("kernel") flags(no-parse)); EOF ;; GNU/kFreeBSD) -- 1.7.10.4

2 1

Embedded log path gives syntax error
by Anton Koldaev 06 Dec '12

06 Dec '12

https://gist.github.com/40c233f40ebc8bc9163e -- Best regards, Koldaev Anton

2 1

openSUSE packages updated, SLES 11 SP2 added
by Peter Czanik 06 Dec '12

06 Dec '12

Hello, My openSUSE packages are now updated. There were two major changes: - binaries moved from / to /usr (supported by openSUSE 12.1+ and SLES 11 SP2) - lot of legacy ifdefs removed, so old releases ( <12.1, unsupported by the openSUSE project) are not supported any more This has two advantages: a lot cleaner spec file, and SLES support added. SLES now also has SSL support. The SLES package was tested only on SLES 11 SP2, but will most likely also work on older SLES 11 releases as long as /usr is not a separate partition. As usual, my 3.3 packages are available at https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 Once 3.4 beta is released, I'll rework my 3.4 packages too. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Which way is better/faster
by Anton Koldaev 06 Dec '12

06 Dec '12

Let's imagine we have one source with some logs: 80% - normal messages (all messages) 20% - specific messages (some complex regex) Normal messages filter includes specific messages. We need to put these messages into separate files. Which way will give more performance? 1. log(source(s),filter(non-specific),destination(normal),flags(final)) log(source(s),destination(specific),flags(final)) OR 2. log(source(s),filter(specific),destination(specific)) log(source(s),filter(normal),destination(normal),flags(final)) OR 3. log(source(s),filter(normal),destination(normal)) log(source(s),filter(specific),destination(specific),flags(final)) -- Best regards, Koldaev Anton

2 1

Embedded log statements performance
by Anton Koldaev 05 Dec '12

05 Dec '12

Which way is faster: 1. log { source(s_localhost); message("example1"); destination(d_file1); flags(final); log {message("example2"); destination(d_file2); }; }; OR 2. log { source(s_localhost); message("example1"); destination(d_file1); flags(final);}; log { source(s_localhost); message("example2"); destination(d_file2); flags(final);}; -- Best regards, Koldaev Anton

1 0