February 2011 - syslog-ng - lists.balabit.hu

[PATCH] systemd integration, take two
by Gergely Nagy 15 Feb '11

15 Feb '11

Hi! After a short session with Bazsi the other day, the earlier systemd integration patches were reworked slightly. Bazsi refactored the patch, I cleaned it up a little afterwards, and tested it to the best of my knowledge (also thanks to Peter Czanik for testing the new patches aswell). The patches are available on the integration/systemd branch of my repositories: git://git.balabit.hu/algernon/syslog-ng-3.3.git and git://git.balabit.hu/algernon/syslog-ng-3.2.git (for 3.3 and 3.2, respectively). The change in this patchset is that the way systemd sockets are recognised has been reworked, which made my previous GSockAddr changes unnecessary (and thus, those changes were dropped). As a side effect, unix-stream sockets should be working too, alongside of unix-dgram, and extending this to other socket types should be a lot easier than before. I'd like to call for some more testing, and if all goes well, then this version could be merged to the official tree. -- |8]

2 1

syslog-ng Insider - February 2011
by Peter Czanik 11 Feb '11

11 Feb '11

Dear syslog-ng users, This is the first issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com FEATURED NEWS BalaBit has been invited to the CEE Board ----------------------------------------- Common Event Expression (CEE) for Event Interoperability is a US govermental task force aiming to standardize the way computer events are described, logged, and exchanged. A part of CEE will define a generic dictionary for event fields, which will be handy for the syslog-ng patterndb project. More information on CEE: http://cee.mitre.org/about.html Current members of CEE Board: http://cee.mitre.org/board.html Related blog posts: * http://bazsi.blogs.balabit.com/2010/11/patterndb-goes-cee/ * http://czanik.blogs.balabit.com/2010/11/cee/ FOSDEM - Free and Open Source Developers European Meeting ---------------------------------------------------------- Peter Czanik from the syslog-ng upstream attended the conference to help include the most recent syslog-ng versions into the main distributions. Read his blog post at http://czanik.blogs.balabit.com/2011/02/fosdem/ Support the syslog-ng project with your testimonial! ---------------------------------------------------- We have launched a public site for collecting user feedbacks and testimonials. Please help us to improve syslog-ng and tell us your opinion (both negative and positive), experience, feature request, or why you use syslog-ng at http://www.balabit.com/testimonials/send OTHER SHORT NEWS * A new syslog-ng driver for the mongodb() NOSQL database is available in 3.3 alpha1: http://asylum.madhouse-project.org/projects/syslog-ng/mongodb/ * systemd support for recent operating systems: https://lists.balabit.hu/pipermail/syslog-ng/2011-February/015876.html * Debian 6.0 repositories include syslog-ng OSE 3.1: http://packages.debian.org/search?keywords=syslog-ng&searchon=names&suite=s… NEW RELEASES OF THE SYSLOG-NG PRODUCT FAMILY * syslog-ng OSE 3.3 alpha1 - http://bazsi.blogs.balabit.com/2010/12/threading-epoll-on-3-3-mainline/ * syslog-ng PE 4.0 - http://attila.blogs.balabit.com/2011/02/balabit%e2%80%99s-syslog-ng-premium… RECENT WHITEPAPERS * Message Classification with syslog-ng (Free download) http://www.balabit.com/whitepapers/log-classification * Performance measurements of syslog-ng Premium Edition 4.0 (Requires registration at balabit.com) http://www.balabit.com/support/documentation/syslog-ng-pe-v4.0-whitepaper-p… * PCI compliance and forensics in auditing remote server access (Free download) http://www.balabit.com/sites/default/files/scb-v3.0-whitepaper-pci-complian…

2 1

Syslog-ng Windows Agent & WIN2008 Event Forwarding Subscription
by Szilárd Szabó 08 Feb '11

08 Feb '11

Üdv mindenkinek, Van egy kis problémám. Adott egy Windows Server 2008 melyen Event Forwarding Subscription van beállítva a következőek szerint: http://blogs.technet.com/b/wincat/archive/2008/08/11/quick-and-dirty-large-… Az hozzáadtam pár klienst. Az események megérkeznek a Windows Server 2008-ra hiba nélkül a ForwardedEvents -be. A Problémám a következő: A Windows Server 2008-ra telepítettem egy Syslog-ng Windows Agent 3.2.1 verziót, és beállítottam a log továbbítást egy Syslog-ng PE-felé. Az események megérkeznek, de felettéb érdekesen :) Jan 20 16:06:34 COMPUTER1 NT: AUTHORITY\ANONYMOUS LOGON: ForwardedEvents Security: [] (EventID 538) Jan 20 16:06:34 COMPUTER2 NT: AUTHORITY\ANONYMOUS LOGON: ForwardedEvents Security: [] (EventID 538) Jan 20 16:06:34 COMPUTER3 NT: AUTHORITY\ANONYMOUS LOGON: ForwardedEvents Security: [] (EventID 538) A Windows 2008 továbbá 64bites, tehát AD-ból van menedzselve. Ez most Agent probléma lehet, vagy a Event Forwarding Subscription-nal van a baj? Üdv Szilárd

5 11

[RFC] value-pairs(), take #3
by Gergely Nagy 07 Feb '11

07 Feb '11

Hi! Based on the feedback from this list, we've had a little discussion with Bazsi on how to improve value-pairs(), and we came up with something that is hopefully more consistent and easier to use than my last proposal. The Syntax ========== We'd have two syntaxes, one for the configuration file itself (usable by the drivers), and one for template functions (eg, tfjson): they'll share most properties, the difference will be in how they appear. See the example below: config file: ------------ value-pairs( scope(nv_pairs core syslog all_macros selected_macros everything) exclude("R_*") exclude("S_*") key(".SDATA.meta.sequenceId") pair("MSGHDR" "$PROGRAM[$PID]: ") ) template function: ------------------ $(format-json --scope nv_pairs,core,syslog,all_macros,selected_macros,everything \ --exclude R_* --exclude S_* --key .SDATA.meta.sequenceId \ --pair MSGHDR="$PROGRAM[$PID]: ") Explanation ----------- The above examples would start with a full set of name-value pairs (due to having "everything" in the scope; we could start with selected_macros instead [see below]). The scope can only be extended by subsequent calls to scope(), but even then, the set will be built only once, at the beginning. We'll likely end up with throwing a syntax error during parse if more than one scope() statement is seen, or if it's not the first statement within value-pairs(). However, explicitly specifying a key-value pair (either via key() or pairs()) will use the full set, regardless what scope() was selected. This, however, might change, if people find this too confusing. But changing this will complicate the code quite a lot, and remove some of the flexibility. >From this set, we exclude every pair where the key begins with "R_" or "S_", then we explicitly include .SDATA.meta.sequenceId (though, in this example, this is useless, as it's already included due to the scope, and wasn't excluded). Then add a custom key-value pair. Syntax Details -------------- The starting name-value pair set will be defined by the scope() statement, which can have the following values: * nv_pairs: The name-value pair database, including some frequently used builtins (currently: HOST, HOST_FROM, MESSAGE, PROGRAM, PID, MSGID, SOURCE and LEGACY_MSGHDR) * rfc3164, alias core, alias base: The basic pairs from RFC3164: $FACILITY, $SEVERITY (= $LEVEL), $DATE(=$S_DATE), $HOST, $PROGRAM, $PID, and $MSG. * rfc5424, alias syslog: The pairs from rfc3164 plus $SDATA and $MSGID. * all_macros: All macros known to syslog-ng (including all of the above, pretty much) * selected_macros: rfc5424 + $TAGS, $SOURCEIP, $SEQNUM * everything: all of the above, combined Each key is added to the set only once, naturally. scope() was introduced as a replacement for builtins(), which was unclear and inflexible. scope() does the job far better, and is - in my opinion - a lot clearer too. Apart from scope(), we have a few more statements: * select() / exclude(): We wanted to rename select() to include(), but syslog-ng already has an include() statement, and I ran into problems during the rename. It's undecided whether we'll remain with select() or adjust the parser to treat the two include statements differently (I'd opt for select()). The difference between the previous implementation's select()/exclude() is that in the new implementation, the first match will matter. This gets rid of the confusing priority stuff, and is still flexible enough (especially with the introduction of scope()) for all cases we could come up with. * key(): One can list macros with this statement. It does the same thing "$HOST" and friends did in the previous implementation, one just needs to use a statement this time, for clarity's sake. * pairs(): Same thing as the previous implementation's ("key" "value") construct. Current shortcomings: * List separation: at the moment, list values need to be space separated, and the key-value pairs (see pairs()) need a space separator too. In the long run, we'd like to allow commas as separators too. Another example --------------- value-pairs( scope(selected_macros nv_pairs) select(.*) select("usracct.*") select("secevt.*") select(".SDATA.*") exclude("*") key("SEVERITY") key("HOST") key("PROGRAM") key("PID") key("MSG") key("TAGS") pair("timestamp" "$UNIXTIME") ); This will start with a base set of selected_macros and nv_pairs, select a few specified patterns, and exclude everything else. Then it will explicitly add a few keys (which does not need to be part of the original set!), and a custom key-value pair. I hope this was understandable, and better than the previous proposal. As soon as I start working on implementing this proposal, the code will be available from the work/value-pairs/base branch of my git tree: git://git.madhouse-project.org/syslog-ng/syslog-ng-3.3.git (or browsable on the web at: http://git.madhouse-project.org/syslog-ng/syslog-ng-3.3/log/?h=work/value-p…) And as always, Your feedback is most appreciated! Nothing is set in stone yet, and I'd love to hear your opinion. -- |8]

3 4

syslog-ng performance tuning
by Zoltán Pallagi 07 Feb '11

07 Feb '11

Hi all, I've just measured the performance of syslog-ng PE and written a blog about syslog-ng performance tuning. Perhaps, it will be useful for OSE, too: http://pzolee.blogs.balabit.com/en/2011/02/syslog-ng-performance-tuning -- pzolee

1 0

syslog-ng af_sql driver experiencing segfault
by Matthew Hall 07 Feb '11

07 Feb '11

Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xf7b12b90 (LWP 23608)] 0x00c5c7dc in memcpy () from /lib/libc.so.6 (gdb) bt #0 0x00c5c7dc in memcpy () from /lib/libc.so.6 #1 0xf7e21fed in g_string_insert_len () from /home/y/lib/libglib-2.0.so.0 #2 0xf7e223e8 in g_string_append_len () from /home/y/lib/libglib-2.0.so.0 #3 0xf7edc1d6 in result_append (result=0x8cc1780, sstr=0x910ffaf "", len=3160627, escape=0) at templates.c:213 #4 0xf7ede535 in log_template_append_format_with_context (self=0x8ba0098, messages=0xf7b12204, num_messages=1, opts=0x8bb8aec, tz=1, seq_num=945, result=0x8cc1780) at templates.c:1058 #5 0xf7ede83f in log_template_append_format (self=0x8ba0098, lm=0x90d93b0, opts=0x8bb8aec, tz=1, seq_num=945, result=0x8cc1780) at templates.c:1109 #6 0xf7ede898 in log_template_format (self=0x8ba0098, lm=0x90d93b0, opts=0x8bb8aec, tz=1, seq_num=945, result=0x8cc1780) at templates.c:1116 #7 0xf7b459e6 in afsql_dd_insert_db (self=0x8bb8a58) at afsql.c:681 #8 0xf7b45fd9 in afsql_dd_database_thread (arg=0x8bb8a58) at afsql.c:823 #9 0xf7ed1405 in worker_thread_func (st=0x8ba4238) at misc.c:593 #10 0xf7e28c51 in ?? () from /home/y/lib/libglib-2.0.so.0 #11 0x08ba4238 in ?? () #12 0x08ba4248 in ?? () #13 0x00000005 in ?? () #14 0xf7e28b06 in ?? () from /home/y/lib/libglib-2.0.so.0 #15 0x00d352c6 in ?? () from /lib/libpthread.so.0 #16 0x00000000 in ?? () (gdb) Looks like I am running into problems with the database code. I'll investigate but it's going to take a while so I hope somebody else might have an idea. This is 3.2 verion 2010-01-31 git. Thanks, Matthew.

2 2

How to stop repeated messages in Syslog-ng?
by Wagner Pereira 06 Feb '11

06 Feb '11

Hi, folks. I am using Syslog-ng as a Cacti's plugin. There are some messages which had been logged since last months. I'd like to ignore these specific messages, I mean, stop to login them. I tried to turn ON two variables in the php.ini... ignore_repeated_source = On ignore_repeated_errors = On ...but unsuccesfully (at least, I changed them and didn't restart any service...is it required?) Is that possible to stop them? Att, -- Wagner Pereira PoP-SP/RNP - Ponto de Presença da RNP em São Paulo CCE/USP - Centro de Computação Eletrônica da Universidade de São Paulo http://www.pop-sp.rnp.br Tel. +55 11 3091-8901 fone@RNP: 10158902

3 2

Testing structured data syslog messages
by Lance Laursen 06 Feb '11

06 Feb '11

Hello list, I am currently trying to test structured data log messages with syslog-ng. I'm running syslog-ng-3.1.2-1.rhel5. Setup is very simple, with a udp source and a destination to /var/log/messages. Here's the line i'm trying to use to test with, which according to the RFC, should be fine: echo -n '<165>1 2011-02-03T22:14:15.003Z superhostomg process - ID47 [exampleSDID@32473 iut="9" eventSource="rawr" eventID="69"] Message portion. Test log with structured data.' | nc -w 1 -u localhost 514 Unfortunately it is showing up in logs as this: 2011-02-03T22:45:30+00:00 localhost 1 2011-02-03T22:14:15.003Z superhostomg process - ID47 [exampleSDID@32473 iut="9" eventSource="rawr" eventID="69"] Message portion. Test log with structured data. So all of the metadata is being printed to $MSG. What am I doing wrong? Thanks,

4 4

Enhancement Request For program(...) Destination
by Matthew Hall 06 Feb '11

06 Feb '11

Could the program dest driver be enhanced to allow specifying the working directory? This would simplify the process of making movable software.

2 1

[PATCH] support for older PCRE
by Matthew Hall 06 Feb '11

06 Feb '11

Can we get this one added for older PCRE support? Bazsi made the old version, I tested and I updated it to apply cleanly on 3.2 master. Signed-off-by: Matthew Hall <mhall(a)mhcomputing.net> --- diff --git a/configure.in b/configure.in index 0c6cef2..f574f48 100644 --- a/configure.in +++ b/configure.in @@ -25,6 +25,8 @@ EVTLOG_MIN_VERSION="0.2" OPENSSL_MIN_VERSION="0.9.8" LIBDBI_MIN_VERSION="0.8.0" PCRE_MIN_VERSION="7.3" +dnl mhall allow older pcre +PCRE_MIN_VERSION="6.1" dnl *************************************************************************** dnl Initial setup diff --git a/lib/logmatcher.c b/lib/logmatcher.c index f0a822c..d003c58 100644 --- a/lib/logmatcher.c +++ b/lib/logmatcher.c @@ -520,8 +520,13 @@ log_matcher_pcre_re_compile(LogMatcher *s, const gchar *re) if (self->super.flags & LMF_ICASE) flags |= PCRE_CASELESS; + +/* mhall allow older pcre */ +#ifdef PCRE_NEWLINE_ANYCRLF if (self->super.flags & LMF_NEWLINE) flags |= PCRE_NEWLINE_ANYCRLF; +#endif + if (self->super.flags & LMF_UTF8) { gint support;

2 1