- syslog-ng - lists.balabit.hu

syslog-ng OSE 3.4 Administrator Guide draft update
by Fekete Róbert 07 Dec '12

07 Dec '12

Hi, My colleague, Ancsur Sebestyen had some time this week to work on the 3.4 admin guide a bit. Still far from being finished or up-to-date, the following new stuff has been documented since the first draft: * The tfhash template function * The JSON parser * The --disable-json compile option * The @PCRE@ patterndb parser * Bug 204: dbparser docs not clear for @NUMBER@ * Bug 205 Docs for macro MSGHDR incorrect * The LOGHOST macro * The syslog-ng-ctl reload option is included in the syslog-ng-ctl man page Also, the links to the docs have changed. The new ones are: HTML: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… Single-page HTML: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… PDF: https://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.4-gui… As always, any feedback is welcome. Robert Fekete

2 1

[PATCH] system(): Use follow-freq(1) on FreeBSD prior to 9.1
by Gergely Nagy 07 Dec '12

07 Dec '12

FreeBSD prior to 9.1 does not have a kqueue-pollable /dev/klog, so syslog-ng would fall back to using timers, except that follow-freq() is explicitly set to zero, also diabling timer-based polling. This has the effect of making syslog-ng not read from /dev/klog at all. To work around the issue, on FreeBSD prior to 9.1, use follow-freq(1) for /dev/klog when using the system() source. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- scl/system/generate-system-source.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/scl/system/generate-system-source.sh b/scl/system/generate-system-source.sh index 708a7a4..34cc009 100755 --- a/scl/system/generate-system-source.sh +++ b/scl/system/generate-system-source.sh @@ -74,7 +74,17 @@ EOF cat <<EOF unix-dgram("/var/run/log"); unix-dgram("/var/run/logpriv" perm(0600)); -file("/dev/klog" follow-freq(0) program-override("kernel") flags(no-parse)); +EOF + case "$osversion" in + 7.*|8.*|9.0*) + ffreq=1 + ;; + *) + ffreq=0 + ;; + esac + cat <<EOF +file("/dev/klog" follow-freq($ffreq) program-override("kernel") flags(no-parse)); EOF ;; GNU/kFreeBSD) -- 1.7.10.4

2 1

Embedded log path gives syntax error
by Anton Koldaev 06 Dec '12

06 Dec '12

https://gist.github.com/40c233f40ebc8bc9163e -- Best regards, Koldaev Anton

2 1

openSUSE packages updated, SLES 11 SP2 added
by Peter Czanik 06 Dec '12

06 Dec '12

Hello, My openSUSE packages are now updated. There were two major changes: - binaries moved from / to /usr (supported by openSUSE 12.1+ and SLES 11 SP2) - lot of legacy ifdefs removed, so old releases ( <12.1, unsupported by the openSUSE project) are not supported any more This has two advantages: a lot cleaner spec file, and SLES support added. SLES now also has SSL support. The SLES package was tested only on SLES 11 SP2, but will most likely also work on older SLES 11 releases as long as /usr is not a separate partition. As usual, my 3.3 packages are available at https://build.opensuse.org/project/show?project=home%3Aczanik%3Asyslog-ng33 Once 3.4 beta is released, I'll rework my 3.4 packages too. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Which way is better/faster
by Anton Koldaev 06 Dec '12

06 Dec '12

Let's imagine we have one source with some logs: 80% - normal messages (all messages) 20% - specific messages (some complex regex) Normal messages filter includes specific messages. We need to put these messages into separate files. Which way will give more performance? 1. log(source(s),filter(non-specific),destination(normal),flags(final)) log(source(s),destination(specific),flags(final)) OR 2. log(source(s),filter(specific),destination(specific)) log(source(s),filter(normal),destination(normal),flags(final)) OR 3. log(source(s),filter(normal),destination(normal)) log(source(s),filter(specific),destination(specific),flags(final)) -- Best regards, Koldaev Anton

2 1

Embedded log statements performance
by Anton Koldaev 05 Dec '12

05 Dec '12

Which way is faster: 1. log { source(s_localhost); message("example1"); destination(d_file1); flags(final); log {message("example2"); destination(d_file2); }; }; OR 2. log { source(s_localhost); message("example1"); destination(d_file1); flags(final);}; log { source(s_localhost); message("example2"); destination(d_file2); flags(final);}; -- Best regards, Koldaev Anton

1 0

Log messages format difference between version 2.0.9 and 3.3.4
by Alptugay Değirmencioğlu 05 Dec '12

05 Dec '12

Hello all. We are using syslog-ng to transmit messages to a stream and an application then reads the messages from the stream and parses them. We have been using 2.0.9 normally. But when we updated to 3.3.4 the message format coming to the stream has changed. Btw we are using CentOS 5 with kernerl version 3.5.3 In 2.0.9 we had the following config: ----------- destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSG\n") template-escape(yes) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; were receiving these kinds of messages: 2012-12-05\\01110:40:13\\011ogm42\\011dansguardian[21839]: 2012.12.5-10:40:13\\011-\\01110.42.1.208\\011http://i.radikal.com.tr/90x70/2012/12/05/fft68_mf1227270.Jpeg\\011*EXCEPTION*Ayricalikli_kullanici_adina_sahipsiniz.\\011GET\\0112499\\0110\\011\\0111\\011200\\011-\\011\\011\\011-\\011 18 You can see that \\011 was a kind of delimeter to us. ----------- And in the new syslog-ng version we have the following config (with 4 back slash \\\\, $MSG changed to $MSGHDR$MSGONLY and template-escape changed from yes to no): destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n") template-escape(no) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (The first parts are okay. But the \\011 int the $MSGONLY have become \t): 2012-12-05\\01110:45:24\\011localhost\\011dansguardian[21135]: 2012.12.5-10:45:24\t-\t127.0.0.1\thttp://www.ipnedir.com\t*SCANNED*\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t 76 ------------ With this config: destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\\\011$HOUR:$MIN:$SEC\\\\011$HOST\\\\011$MSGHDR$MSGONLY\n") template-escape(no) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (The first parts are okay. But the \\011 int the $MSGONLY are lost) 2012-12-05\\01110:48:33\\011localhost\\011dansguardian[21135]: 2012.12.5-10:48:33-127.0.0.1http://www.ipnedir.comGET361601200text/html- 73 ----------------- With this config: destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n") template-escape(no) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (Now the output has become completely wrong) 2012-12-0501110:52:06011localhost011dansguardian[21135]: 2012.12.5-10:52:06\t-\t127.0.0.1\thttp://www.ipnedir.com\t\tGET\t3616\t0\t\t1\t200\ttext/html\t\t\t-\t 89 --------------------- And finally with this config: destination webfilter { unix-stream("/var/log/streams/webfilter" template("$YEAR-$MONTH-$DAY\\011$HOUR:$MIN:$SEC\\011$HOST\\011$MSGHDR$MSGONLY\n") template-escape(yes) ); }; filter f_webfilter { match("dansguardian\\[.*\\]: .*\t"); }; log { source(s_all); filter(f_webfilter); destination(webfilter); }; We get the following output (Note that this config is the same config with the one we used in 2.0.9 except the $MSGHDR$MSGONLY part ) 2012-12-0501110:53:58011localhost011dansguardian[21135]: 2012.12.5-10:53:58-127.0.0.1http://www.ipnedir.comGET361601200text/html- 184 --------------------------- So what we want to accomplish is that we want the old output back with the new syslog :) How can we achieve this? Btw when restarting syslog-ng service we get the following messages. I don't know if it has anything to do with this. Restarting syslog-ng: Error opening plugin module; module='dbparser', error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='log_prefix', change='program_override' WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' Shutting down syslog-ng: [ OK ] Starting syslog-ng: Error opening plugin module; module='dbparser', error='/usr/lib/syslog-ng/libdbparser.so: undefined symbol: evt_tag_long' Your configuration file uses an obsoleted keyword, please update your configuration; keyword='log_prefix', change='program_override' WARNING: the match() filter without the use of the value() option is deprecated and hinders performance, please update your configuration; Your configuration file uses an obsoleted keyword, please update your configuration; keyword='sync', change='flush_lines' WARNING: window sizing for tcp sources were changed in syslog-ng 3.3, the configuration value was divided by the value of max-connections(). The result was too small, clamping to 100 entries. Ensure you have a proper log_fifo_size setting to avoid message loss.; orig_log_iw_size='10', new_log_iw_size='100', min_log_fifo_size='10000' [ OK ]

2 1

writing error to a mounted drive
by Frank Scalzo 03 Dec '12

03 Dec '12

: Error opening file for writing; filename='/var/log/hosts/XXXX/XXXXX', error='Permission denied (13)' I made the directory hosts inside /var/log/ and before I mounted the drive to the path /var/log/hosts files would write just fine but when I mounted the drive in that path I get the permissions denied error. running on redhat REHL 6and syslog-NG 3.2 OSE

2 1

[PATCH] [patterndb] fix a missing XML semantic check in PDBLoader
by balint.kovacs＠balabit.com 03 Dec '12

03 Dec '12

From: Balint Kovacs <blint(a)blint.hu> pattnerndb has failed to check if we are within a rule when trying to add tags and values, so a patterndb xml containing these elements outside of a rule element would cause syslog-ng (and pdbtool) to segfault Signed-off-by: Balint Kovacs <blint(a)blint.hu> --- modules/dbparser/patterndb.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/modules/dbparser/patterndb.c b/modules/dbparser/patterndb.c index 004d6b7..d2349f6 100644 --- a/modules/dbparser/patterndb.c +++ b/modules/dbparser/patterndb.c @@ -1154,10 +1154,20 @@ pdb_loader_text(GMarkupParseContext *context, const gchar *text, gsize text_len, } else if (state->in_tag) { + if (!state->in_rule) + { + *error = g_error_new(1, 0, "Unexpected <tag> element, must be within a rule"); + return; + } pdb_message_add_tag(state->current_message, text); } else if (state->value_name) { + if (!state->in_rule) + { + *error = g_error_new(1, 0, "Unexpected <value> element, must be within a rule"); + return; + } if (!state->current_message->values) state->current_message->values = g_ptr_array_new(); -- 1.7.9.5

2 1

syslog uses wrong and weird hostnames
by Daniel Neubacher 28 Nov '12

28 Nov '12

Hello there, My syslog-ng is logging fine from 700 servers but a few times per day it is logging into a wrong folder which is created by the $HOST variable. Normally syslog should use a hostname like host-1.worker.foobar.com but a few times a day it creates a folder names like: 4.22 p6-allrouter? p6-localnet p6-mc? p6-mcastpref p6-mcastpref? host-1.worker.foobar.xcom and other weird names... It only logs one or two lines and then uses the right folder again. Does anyone have a clue where I have to look for the problem? Here are the option line from the server and the client: Client: @version: 3.3 options { threaded(yes); use_dns(yes); use_fqdn(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(10); log_msg_size(256000); log_fifo_size(100000); normalize_hostnames(yes); check_hostname(yes); bad_hostname("^gconfd$"); create_dirs(yes); owner("root"); group("root"); perm(0640); time_reopen(30); }; Server: options { threaded(yes); owner("root"); group("root"); perm(0660); dir_owner("root"); dir_group("root"); dir_perm(0770); create_dirs(yes); chain_hostnames(no); normalize_hostnames(yes); check_hostname(yes); keep_hostname(no); use_fqdn(yes); dns_cache(yes); dns_cache_size(16384); dns_cache_expire(3600); dns_cache_expire_failed(60); log_msg_size(256000); log_fifo_size(1000000); }; Destination example: destination d_syslog { file("/log/syslog/${R_YEAR}/${R_MONTH}/${R_DAY}/$HOST/$PROGRAM"); }; -- Daniel Neubacher, Network Administrator daniel.neubacher(a)xing.com<mailto:daniel.neubacher@xing.com> XING AG Gaensemarkt 43, 20354 Hamburg, Germany Tel. +49 40 419131-28, Fax +49 40 419131-11 Commercial Reg. (Registergericht): Amtsgericht Hamburg, HRB 98807 Exec. Board (Vorstand): Dr. Stefan Groß-Selbeck (Vorsitzender), Dr. Thomas Vollmoeller, Ingo Chu, Dr. Helmut Becker, Jens Pape Chairman of the Supervisory Board (Aufsichtsratsvorsitzender): Dr. Neil Sunderland This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorised copying, disclosure or distribution of the material in this e-mail is strictly forbidden and may be unlawful.

2 3