- syslog-ng - lists.balabit.hu

[PATCH] afmongodb: Fix threaded operation.
by Gergely Nagy 02 Sep '11

02 Sep '11

Ported a patch from afsql to afmongodb, that makes the writer thread wake up only when it's synced. This makes threaded(yes) work more reliably. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afmongodb/afmongodb.c | 18 ++++++++++++++++-- 1 files changed, 16 insertions(+), 2 deletions(-) diff --git a/modules/afmongodb/afmongodb.c b/modules/afmongodb/afmongodb.c index 9e155ed..0f01f05 100644 --- a/modules/afmongodb/afmongodb.c +++ b/modules/afmongodb/afmongodb.c @@ -497,6 +497,17 @@ afmongodb_dd_free(LogPipe *d) } static void +afmongodb_dd_queue_notify(gpointer user_data) +{ + MongoDBDestDriver *self = (MongoDBDestDriver *)user_data; + + g_mutex_lock(self->queue_mutex); + g_cond_signal(self->writer_thread_wakeup_cond); + log_queue_reset_parallel_push(self->queue); + g_mutex_unlock(self->queue_mutex); +} + +static void afmongodb_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_options, gpointer user_data) { MongoDBDestDriver *self = (MongoDBDestDriver *)s; @@ -509,12 +520,15 @@ afmongodb_dd_queue(LogPipe *s, LogMessage *msg, const LogPathOptions *path_optio g_mutex_lock(self->queue_mutex); self->last_msg_stamp = cached_g_current_time_sec (); queue_was_empty = log_queue_get_length(self->queue) == 0; - log_queue_push_tail(self->queue, msg, path_options); g_mutex_unlock(self->queue_mutex); + + log_queue_push_tail(self->queue, msg, path_options); g_mutex_lock(self->suspend_mutex); if (queue_was_empty && !self->writer_thread_suspended) - g_cond_signal(self->writer_thread_wakeup_cond); + { + log_queue_set_parallel_push(self->queue, 1, afmongodb_dd_queue_notify, self, NULL); + } g_mutex_unlock(self->suspend_mutex); } -- 1.7.0.4

2 1

log for patterns
by Peter Czanik 02 Sep '11

02 Sep '11

Hello, While my previous post was about Windows, we did not forget about our UNIX/Linux users :) I'd like to extend our pattern database with new patterns. So I have some related questions: - Until now we dealt mostly with login/logut events of different applications, as this is one of the most interesting information on servers. Is it OK with you, or we should cover also other events? (Which?) - Which applications are you mostly interested in? - Creating logs in a "lab" environment just for pattern creation is very time consuming. It would be very helpful for us, if you could send logs in exchange for patterns. Just make sure, that there is no sensitive data left in the logs, as the result will be published to make it available for the whole syslog-ng community. I published a blog ( http://czanik.blogs.balabit.com/2010/11/log-sample-collecting-project/ ), how I collect logs for pattern creation, but of course, any logs are welcome! Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

4 3

License for syslog-ng OSE v 3.2.x
by Yu Watanabe 01 Sep '11

01 Sep '11

Hello all. I would like to ask question for the license of syslog-ng. syslog-ng is licensed under the combination of the GPL and LGPL licenses. The syslog-ng core (src/ subdirectory) is licensed under the Lesser General Public License as described in the file LGPL.txt The syslog-ng modules (src/xxxx/ subdirectories) are licensed under the General Public License as described in the file GPL.txt Would it be correct to understand as below? LGPL = LGPLv2.1 GPL = GNU General Public License, version 1 I couldn't find the both text in the tarball so asked this question. Thanks! Yu

3 3

[PATCH] [pdbtool] debug-id option
by Balint Kovacs 31 Aug '11

31 Aug '11

Hi, While working with `pdbtool test`, I found that it's quite uncomfortable to find problems with non-matching messages, as it only displays the ID of the erroneous pattern. Right now you would need to copy the example message and program name, and pass it to `pdbtool match` as arguments to find out what's the exact issue and get a nice colorized output pointing to the problematic part of the pattern. To make that easier, I've done a small enhancement to pdbtool, an option to do a full, colorized debug output on a given rule by only supplying its ID. This would look something like (coloring lost in email): blint@lyra:~/blah/syslog-ng-ose-mainline-3.4$ ./bin/pdbtool debug-id -p /var/tmp/patterndb/system-bind.xml -r "b57a384f-c8be-41e9-bc10-735695dc63e7" Pattern matching part: unexpected RCODE (REFUSED) resolving @QSTRING:.dict.arpa=hushmail.com/AAAA/IN@:@QSTRING:.dict.src=203.197.12.30@deliberately freaked up test message 53 Matching part: unexpected RCODE (REFUSED) resolving 'hushmail.com/AAAA/IN': 203.197.12.30# Values: MESSAGE=unexpected RCODE (REFUSED) resolving 'hushmail.com/AAAA/IN': 203.197.12.30#deliberately freaked up test message 53 PROGRAM=named .classifier.class=unknown The patch is against 3.4, but should apply to 3.3 as well. Balint commit 93ca04700f8706643fedea51936af02daa314766 Author: Balint Kovacs<blint(a)balabit.hu> Date: Mon Aug 15 11:07:50 2011 +0200 [pdbtool] Implemented debug-id option Added a new option to pdbtool to test a specific rule against its example message with colorizing debug turned on. Signed-off-by: Balint Kovacs<blint(a)balabit.hu> diff --git a/modules/dbparser/pdbtool.c b/modules/dbparser/pdbtool.c index 64ccd17..f64012d 100644 --- a/modules/dbparser/pdbtool.c +++ b/modules/dbparser/pdbtool.c @@ -640,6 +640,75 @@ static GOptionEntry match_options[] = { NULL, 0, 0, G_OPTION_ARG_NONE, NULL, NULL } }; +static gchar *rule_id = NULL; + +static gboolean +pdbtool_debug_id(int argc, char *argv[]) +{ + PatternDB *patterndb; + PDBExample *example; + GList *examples = NULL; + gboolean id_is_found = FALSE; + + debug_pattern = TRUE; + debug_pattern_parse = FALSE; + color_out = TRUE; + colors = full_colors; + + if (!rule_id) + { + printf("Please specify a rule ID to be tested against its example message!\n"); + return FALSE; + } + + patterndb = pattern_db_new(); + if (!pdb_rule_set_load(patterndb->ruleset, configuration, patterndb_file,&examples)) + { + pattern_db_free(patterndb); + return FALSE; + } + + while (examples) + { + example = examples->data; + + if (strcmp(example->rule->rule_id, rule_id) != 0) + { + examples = g_list_delete_link(examples, examples); + continue; + } + + id_is_found = TRUE; + + if (example->message&& example->program) + { + match_message = example->message; + match_program = example-> program; + pdbtool_match(argc, argv); + } + examples = g_list_delete_link(examples, examples); + } + + pattern_db_free(patterndb); + + if (!id_is_found) + { + printf("Could not find a corresponding ID in the patterndb file or the rule does not have an example message.\n"); + return FALSE; + } + + return TRUE; +} + +static GOptionEntry debug_id_options[] = +{ + { "pdb", 'p', 0, G_OPTION_ARG_STRING,&patterndb_file, + "Name of the patterndb file", "<patterndb_file>" }, + { "ruleid", 'r', 0, G_OPTION_ARG_STRING,&rule_id, + "ID of the patterndb rule to debug", "<rule_id>" }, + { NULL, 0, 0, G_OPTION_ARG_NONE, NULL, NULL } +}; + static gboolean test_validate = FALSE; static gboolean @@ -934,6 +1003,7 @@ static struct { "dump", dump_options, "Dump pattern datebase tree", pdbtool_dump }, { "merge", merge_options, "Merge pattern databases", pdbtool_merge }, { "test", test_options, "Test pattern databases", pdbtool_test }, + { "debug-id", debug_id_options, "Test pattern databases", pdbtool_debug_id }, { "patternize", patternize_options, "Create a pattern database from logs", pdbtool_patternize }, { NULL, NULL }, };

2 7

Duplicate Hostnames entries in syslog-ng
by Swati4 M/TVM/TCS 30 Aug '11

30 Aug '11

Hi, My client machine is running syslog-ng 3.0 version and remote server is running syslog. On logging messages from client to server,I am encountering duplicate entries for hostnames in the remote syslog messages and an extra colon. In local log files running syslog-ng,there are no duplicate entries. Have tried the following options,but no luck:- keep_hostname(yes); chain_hostnames(no); long_hostnames(no); For "extra colon" seen in remote messages,I have tried attaching some templates to filter my messages. Since,I guess in syslog-ng 3.0 the macro $MSG has been changed to $MSGHDR. Remote Output :- Aug 12 03:42:49 base0-0-0-1-0-11-1 base0-0-0-1-0-11-1 kernel: : device eth0 left promiscuous mode Expected Output :- Aug 12 03:42:49 base0-0-0-1-0-11-1 kernel:device eth0 left promiscuous mode Can anybody please help me with the issue ? Thanks in advance. Regards This is a snippet of my syslog-ng.conf file :- options { flush_lines (0); time_reopen (10); log_fifo_size (1000); long_hostnames (off); use_dns (yes); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; =====-----=====-----===== Notice: The information contained in this e-mail message and/or attachments to it may contain confidential or privileged information. If you are not the intended recipient, any dissemination, use, review, distribution, printing or copying of the information contained in this e-mail message and/or attachments to it are strictly prohibited. If you have received this communication in error, please notify us by reply e-mail or telephone and immediately and permanently delete the message and any attachments. Thank you

1 0

UDP Dropping packets
by Aldrich, Jamie S 30 Aug '11

30 Aug '11

I am trying to send the output from an application log to a LogLogic device, and it appears we are getting UDP drops repeatedly. Any ideas on what I can do to fix them? Aug 25 15:24:55 lxfwossecp3 syslog-ng[2297]: Log statistics; dropped='udp(10.13.33.11:514)=0', processed='center(queued)=6376', processed='center(received)=6376', processed='destination(d_messages)=6374', processed='destination(d_bo ot)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=1', processed='destination(d_mlal)=0', processed='destination(d_kern)=0', processed='destination(d_mesg)=1', processed='destination(d_cons)=0', processed='des tination(d_spol)=0', processed='destination(d_mail)=0', processed='source(s_sys)=2', processed='source(s_file)=6374', suppressed='udp(10.13.33.11:514)=0' Aug 25 15:34:55 lxfwossecp3 syslog-ng[2297]: Log statistics; dropped='udp(10.13.33.11:514)=0', processed='center(queued)=6378', processed='center(received)=6378', processed='destination(d_messages)=6374', processed='destination(d_bo ot)=0', processed='destination(d_auth)=0', processed='destination(d_cron)=2', processed='destination(d_mlal)=0', processed='destination(d_kern)=0', processed='destination(d_mesg)=2', processed='destination(d_cons)=0', processed='des tination(d_spol)=0', processed='destination(d_mail)=0', processed='source(s_sys)=4', processed='source(s_file)=6374', suppressed='udp(10.13.33.11:514)=0' Jamie

4 6

3.3 git
by Peter Czanik 30 Aug '11

30 Aug '11

Hello, Just tested the current 3.3 git on openSUSE and FreeBSD. I made a 'make dist' and then used the resulting tgz for builds. I also did some quick functionality tests. While file names mention beta2 everywhere, it's actually a git checkout from this morning. Work in progress openSUSE packages for 11.4 and Factory are available at http://download.opensuse.org/repositories/home:/czanik:/syslog-ng33/ (it fails at post build checks on 11.3 and it does not compile on SLES). openSUSE package sources are available at https://build.opensuse.org/package/files?package=syslog-ng&project=home%3Ac… It's almost the same as the beta2 version included in factory, but I could remove all the patches hand picked from git. If you want to build syslog-ng 3.3 git on FreeBSD, you can download the source from the above URL, copy it to /usr/ports/distfiles/syslog-ng_3.3.0beta2.tar.gz and "rm distinfo" from /usr/ports/sysutils/syslog-ng3-devel, as it contains info about the original sources. Bye, -- Peter Czanik (CzP) <czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

malformed syslog packets?
by Matt Zagrabelny 30 Aug '11

30 Aug '11

Hi! I've got a central log server running the OSE 3.1.3 version of syslog-ng: dpkg -l syslog-ng Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Description +++-========================================-========================================-================================================================================================ ii syslog-ng 3.1.3-3 Next generation logging daemon I have dns lookup turned on via: options { use_dns(yes); dns_cache(2000); dns_cache_expire(87600); }; And this seems to work just fine...except for a certain type of device on our network. We have a number of UPSes that log to our central log server and it seems that the dns look ups do not work for those (types of devices). % cd /var/log/syslog-ng/remote_clients % ls -d 10.* 10.25.32.4 10.25.5.15 10.25.5.19 10.25.5.26 10.25.5.35 10.25.5.4 10.25.5.44 10.25.5.51 10.25.5.6 10.25.5.65 10.25.5.69 10.25.5.76 10.25.5.1 10.25.5.16 10.25.5.2 10.25.5.27 10.25.5.36 10.25.5.40 10.25.5.49 10.25.5.52 10.25.5.60 10.25.5.66 10.25.5.7 10.25.5.79 10.25.5.10 10.25.5.17 10.25.5.20 10.25.5.28 10.25.5.37 10.25.5.41 10.25.5.5 10.25.5.55 10.25.5.61 10.25.5.67 10.25.5.72 10.25.5.81 10.25.5.14 10.25.5.18 10.25.5.23 10.25.5.3 10.25.5.38 10.25.5.43 10.25.5.50 10.25.5.58 10.25.5.62 10.25.5.68 10.25.5.75 10.25.5.9 When I look up those IP addresses, they are *all* APC batteries (UPSes). For instance: % dig -x 10.25.5.43 +short kplz246Abat1.d.umn.edu. Is it possible that they are sending some sort of munged data to the log server and syslog-ng is not able to perform the (reverse) name lookup? Any advice? Thanks, much! -Matt Zagrabelny

3 5

syslog-ng.persist file error
by Scott McLane Gardner 29 Aug '11

29 Aug '11

Hello, When I came in this morning, syslog-ng was not running, and the following error is produced when trying to start it with init script: >Error loading persistent configuration file; name='//var/syslog-ng.persist' I'm guessing that additional forward slash is not supposed to be there, but I have no idea how this is getting set or where to change it. Can anyone help me? Thanks, Scott

3 3

[PATCH] templates: Add a C_* family of timestamp macros.
by Gergely Nagy 29 Aug '11

29 Aug '11

The C_* family of timestamp macros work very similarly to R_ (received) and S_ (stamp), except C_ uses the current system time as of the macro expansion. The patch below uses cached_g_current_time() for performance reasons, but in case the minor caching is undesired, it's easy to convert to something else. Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- lib/templates.c | 39 +++++++++++++++++++++++++++++++++++++-- 1 files changed, 37 insertions(+), 2 deletions(-) diff --git a/lib/templates.c b/lib/templates.c index cb0c1b5..63000a3 100644 --- a/lib/templates.c +++ b/lib/templates.c @@ -86,6 +86,7 @@ enum M_RECVD_OFS = M_TIME_MACROS_MAX, M_STAMP_OFS = 2 * M_TIME_MACROS_MAX, + M_CSTAMP_OFS = 3 * M_TIME_MACROS_MAX, }; #define M_TIME_MACROS 15 @@ -171,6 +172,29 @@ LogMacroDef macros[] = { "S_TZ", M_STAMP_OFS + M_TZ }, { "S_UNIXTIME", M_STAMP_OFS + M_UNIXTIME }, + { "C_DATE", M_CSTAMP_OFS + M_DATE }, + { "C_FULLDATE", M_CSTAMP_OFS + M_FULLDATE }, + { "C_ISODATE", M_CSTAMP_OFS + M_ISODATE }, + { "C_STAMP", M_CSTAMP_OFS + M_STAMP }, + { "C_YEAR", M_CSTAMP_OFS + M_YEAR }, + { "C_YEAR_DAY", M_CSTAMP_OFS + M_YEAR_DAY }, + { "C_MONTH", M_CSTAMP_OFS + M_MONTH }, + { "C_MONTH_WEEK", M_CSTAMP_OFS + M_MONTH_WEEK }, + { "C_MONTH_ABBREV", M_CSTAMP_OFS + M_MONTH_ABBREV }, + { "C_MONTH_NAME", M_CSTAMP_OFS + M_MONTH_NAME }, + { "C_DAY", M_CSTAMP_OFS + M_DAY }, + { "C_HOUR", M_CSTAMP_OFS + M_HOUR }, + { "C_MIN", M_CSTAMP_OFS + M_MIN }, + { "C_SEC", M_CSTAMP_OFS + M_SEC }, + { "C_WEEKDAY", M_CSTAMP_OFS + M_WEEK_DAY_ABBREV }, /* deprecated */ + { "C_WEEK_DAY", M_CSTAMP_OFS + M_WEEK_DAY }, + { "C_WEEK_DAY_ABBREV",M_CSTAMP_OFS + M_WEEK_DAY_ABBREV }, + { "C_WEEK_DAY_NAME", M_CSTAMP_OFS + M_WEEK_DAY_NAME }, + { "C_WEEK", M_CSTAMP_OFS + M_WEEK }, + { "C_TZOFFSET", M_CSTAMP_OFS + M_TZOFFSET }, + { "C_TZ", M_CSTAMP_OFS + M_TZ }, + { "C_UNIXTIME", M_CSTAMP_OFS + M_UNIXTIME }, + { "SDATA", M_SDATA }, { "MSGHDR", M_MSGHDR }, { "SOURCEIP", M_SOURCE_IP }, @@ -421,7 +445,7 @@ log_macro_expand(GString *result, gint id, gboolean escape, LogTemplateOptions * gchar buf[64]; gint length; time_t t; - LogStamp *stamp; + LogStamp *stamp, sstamp; glong zone_ofs; if (id >= M_TIME_FIRST && id <= M_TIME_LAST) @@ -438,7 +462,18 @@ log_macro_expand(GString *result, gint id, gboolean escape, LogTemplateOptions * id -= M_STAMP_OFS; stamp = &msg->timestamps[LM_TS_STAMP]; } - else + else if (id >= M_TIME_FIRST + M_CSTAMP_OFS && id <= M_TIME_LAST + M_CSTAMP_OFS) + { + GTimeVal tv; + + id -= M_CSTAMP_OFS; + cached_g_current_time(&tv); + sstamp.tv_sec = tv.tv_sec; + sstamp.tv_usec = tv.tv_usec; + sstamp.zone_offset = -1; + stamp = &sstamp; + } + else { g_assert_not_reached(); break; -- 1.7.0.4

2 1