- syslog-ng - lists.balabit.hu

[PATCH (3.4) 0/2] value-pairs: Efficient sorting
by Gergely Nagy 28 Feb '12

28 Feb '12

In the two patches that will follow, a subtle, yet very important change will be made to the core of value-pairs: instead of building a GHashTable during a value_pairs_foreach(), we now build a binary tree, that's sorted at insert time, and traversing through it is much faster. This also buys us sorted keys almost free of charge! And once we have that, the second patch exports a value_pairs_foreach_sorted() function, so that modules who want to override the default alphabetical sort will be able to do so. -- |8]

2 8

Rewrite backreference oddity
by Ti Leggett 27 Feb '12

27 Feb '12

I have a rewrite rule that is used to replace private IPs that have no DNS name to an internally used name. It looks like: rewrite r_ddn { subst("172.30.60.(\d*)", "d9550-$1.example.com", value("HOST"), type("pcre")); }; For IP 172.30.60.1 I should get a HOST that is d9550-1.example.com, but what I actually get is d9550-.example.com1. Any ideas what I'm doing incorrectly? I've tried encapsulating the back reference in {}. Here's info about my syslog-ng: [root@admin 02]# syslog-ng -V syslog-ng 3.3.4 Installer-Version: 3.3.4 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.3#master#5e44eb46b0d7b86b62f17698e2b6de875ac8d7c6 Compile-Date: Feb 17 2012 10:55:40 Default-Modules: affile,afprog,afsocket,afuser,basicfuncs,csvparser,dbparser,syslogformat Available-Modules: affile,afsocket,dummy,afsocket-tls,afprog,syslogformat,csvparser,afmongodb,confgen,afuser,basicfuncs,dbparser,convertfuncs Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-Linux-Caps: on Enable-Pcre: on

3 7

RFC: value-pairs() improvements, and the way forward
by Gergely Nagy 25 Feb '12

25 Feb '12

Hi! As is my custom when I get stuck thinking about something probably trivial for those more versed in algorithmical thinking, I turn to the list, and dump my ideas, in the hopes to borrow all the good ideas you lot usually have. And unlike most times, I'll try to be short and to the point for a change. Some of you might have heard about this value-pairs thing that found its way into syslog-ng 3.3, and received some neat new features in 3.4, but as a reminder, let me tell you a few words about it: it's a reasonably convenient, simple, yet flexible way to build a set of key-value pairs from all the data syslog-ng managed to collect about and parse out of a message. In 3.4, one can even apply various transformations to the keys, to massage them into a format suitable for applications outside of syslog-ng. As of this writing, the mongodb destination driver and the format-json template function are the two users of this feature, and a few more users will likely see the light of day in the not too distant future. However, there are multiple issues with the current implementation: - It's far slower than it needs to be. - It's hard to sort the keys alphabetically (which would be useful for format-welf, for example, or anywhere else where we want the order of keys to remain constant between messages) - It would be very, very inconvenient to enhance the implementation to be able to handle nested structures. And we do need nested structures: the json parser (in 3.4) already transforms nested structures into an internal format, that could - in theory - be expanded back to the original, or something very close, but value-pairs does not support that. So, I want to change the *implementation* of value-pairs, to fix all of the problems above. The syntax in the config file *will* remain the same (though, both mongodb and format-json will probably grow an extra parameter, that controls whether we want a flat structure, or a nested object structure). As a first step, I will replace the GHashTables (which is a key-value store, optimised for random access, basically) with GTree (which is a sorted key-value store, optimised for ordered traversal, pretty much what we need). This has the advantage of being a bit more efficient than GHashTables, and gaining us an ordered key list for free. The hard part is writing the flat format => tree conversion, and making it efficient. I have some crude code to do it, which iterates through a sorted key list, splits the key up by dots, finds the appropriate node in the tree, if any, and adds a new child node. This means we iterate through the original key list only once (this is good), but we have to perform a lot of lookups in another tree for each key (this is bad). Now, I can optimise this a bit, and use the ordered property of the original list, and store the path to the last inserted node, and try to use that for speedier backtracking. But that's something I'll probably do in a second version of the update.. I believe this method would be reasonably efficient, but if anyone has a better suggestion, please let me know. -- |8]

2 1

[PATCH] afsql: Fix a deadlock when using explicit commits.
by Gergely Nagy 25 Feb '12

25 Feb '12

When using explicit commits, in case the COMMIT results in an error, syslog-ng forgot to release a mutex, resulting in a deadlock the next time the same error happened. The fix is to do release the mutex in this error path, too. Signed-off-by: Fried Zoltan <deirf(a)balabit.hu> Signed-off-by: Patrick Hemmer <syslogng(a)stormcloud9.net> Signed-off-by: Gergely Nagy <algernon(a)balabit.hu> --- modules/afsql/afsql.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/modules/afsql/afsql.c b/modules/afsql/afsql.c index b3fec86..d3c72c9 100644 --- a/modules/afsql/afsql.c +++ b/modules/afsql/afsql.c @@ -890,6 +890,7 @@ afsql_dd_database_thread(gpointer arg) { afsql_dd_disconnect(self); afsql_dd_suspend(self); + g_mutex_unlock(self->db_thread_mutex); continue; } } -- 1.7.9

2 1

help with SDATA
by Chris Johnson 25 Feb '12

25 Feb '12

Based on the following structured syslog, I am trying to extract the reason from the SDATA portion of the log. 2012-02-05T16:24:45.368 RT_FLOW - RT_FLOW_SESSION_CLOSE [junos(a)2636.1.1.1.2.36 reason=\”unset\” ] Problem I think I am running into is the value-pair that I am trying to extract includes the dots so the parser thinks its nested information destination d_mongodb { mongodb( value-pairs( scope("everything") key(".SDATA.junos(a)2636.1.1.1.2.36.reason") ) ); }; Thoughts? Any help is greatly appreciated. Chris

3 4

How to use parser
by 大町靖 25 Feb '12

25 Feb '12

Hello Robert, Thanks for your effort. I rewrite the configurtion below. parser p1{ csv-parser(columns("Msg1","Msg2") delimiters("#AA#") # template ("$Msg1\n$Msg2"); }; destination d_mesge { file("/var/log/messages" template("$MSG1\nMSG2\n")); }; log { source(s_sys); parser(p1); destination(d_mesg); }; But the result isn't unexpected. I want to divide by "#AA#". ("#AA#" is a string.) It seems that the result is divided by the '#' or 'A' . (character). For example, the messages is "12A456 #AA# 789". The result is MSG1 = 12 MSG2=456. I expect that MSG1=12A456 MSG2=789. What should I do ? Thanks, Yasu === Hi, remove the template() option from the parser definition and add it to the definition of the destination. Regards, Robert On 02/07/2012 03:50 PM, 大町靖 wrote: > Hello Experts, > > Please let me know how to use parser setting. > > This is the syslog message. > Msg1 #AA# Msg2 > > I want to divide this message. > I expect the below. > Msg1 > Msg2 > > But It doesn't work well , though I wrote the configuration below. > parser p1{ > csv-parser(columns("Msg1","Msg2") > delimiters("#AA#") > template ("$Msg1\n$Msg2"); > }; > log { source(s_sys); parser(p1); destination(d_mesg); }; > > I check the output. The result wasn't unexpected. > Msg1 #AA# Msg2 > > If my configuration is wrong, please let me know. > > Thanks, > Yasu > > ______________________________________________________________________________ > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng > FAQ: http://www.balabit.com/wiki/syslog-ng-faq > >

2 1

Re: [syslog-ng] Installing syslog-ng 3.3.4 OSS on multiple Centos servers
by Srinivasan Subramanian 25 Feb '12

25 Feb '12

What about Mageia ? I see the latest binaries and source RPMS in the Mageia repository? Looks like the latest will always be available there. Srini > Date: Fri, 24 Feb 2012 14:22:46 +0100 > From: Gergely Nagy <algernon(a)balabit.hu> > Subject: Re: [syslog-ng] Installing syslog-ng 3.3.4 OSS on multiple > Centos servers > To: syslog-ng(a)lists.balabit.hu > Message-ID: <87pqd49xi1.fsf(a)luthien.mhp> > Content-Type: text/plain > > Srinivasan Subramanian <ssrini_vasan(a)hotmail.com> writes: > > > I have a need to install syslog-ng 3.3.4 on multiple Centos servers. > > Unfortunately i am unable to find RPMs for the newer versions and only > > find the source downloads. > > I don't know of any Centos RPMs, but you might try the sRPMs from > SuSE[1]. At worst, they'll be a reasonable starting point. > > [1]: http://software.opensuse.org/search/download?base=openSUSE%3AFactory&file=B… > > --

1 0

3.3.4 anyone else notice a memory leak
by Evan Rempel 25 Feb '12

25 Feb '12

I run a number of syslog-ng processes, and I am noticing that the processes that read from a pipe, and write to a program destination all seem to have a memory leak in them. The rate of leak seems to be dependent on the volume of log lines (not surprising). Has anyone else noticed this. My config is; @version: 3.3 # # UVic syslog-ng configuration options { log_fifo_size(100000); use_fqdn(yes); keep_hostname(yes); chain_hostnames(yes); time_reap(60); time_reopen(5); flush_lines(1000); flush_timeout(1000); }; source local { pipe("/var/log/syslog.pipes/syslogstats" log_iw_size(90000) log_fetch_limit(500) flags(no-parse)); }; source int { internal(); }; destination localsyslog { tcp("localhost" port(1514) localip(localhost) log_fifo_size(50000) template("<$PRI>$S_DATE syslogstats@$HOST syslogstats: $MSGONLY\n") template_escape(no) ); }; destination syslog_stats { program("/home/sysprog/syslog-apps/syslog_stats/syslogstats" template("$MESSAGE\n") template_escape(no) ); }; log { source(local); destination(syslog_stats); flags(flow-control); }; log { source(int); destination(localsyslog); };

6 12

syslog-ng Insider - February 2012
by Peter Czanik 23 Feb '12

23 Feb '12

Dear syslog-ng users, This is the 11th issue of the syslog-ng Insider, a monthly newsletter that brings you syslog-ng related news. Your feedback and news tips about the next issue is welcome at documentation(a)balabit.com <mailto:documentation@balabit.com> FEATURED NEWS Brno: Fedora, CEE, journal and syslog-ng ---------------------------------------- Last week the Brno Red Hat office hosted two conferences, one small about logging and the Fedora Developer conference. The logging miniconf covered some very hot topics: CEE, journal, auditd and some lesser known projects, like ELAPI. After the formal program, we had some very good discussions about the future of logging. You can check the diagram drawn up as conclusion here: https://czanik.blogs.balabit.com/2012/02/brno-fedora-cee-journal-and-syslog… And read more about how syslog-ng supports CEE: http://algernon.blogs.balabit.com/2012/02/cee-handling-with-syslog-ng/ BalaBit has just released the latest version of its leading log management tool, syslog-ng 4 F2 ------------------------------- Adding to the existing, rich feature set which includes high-performance multi-thread processing, encrypted and timestamped log files, disk-based buffering, direct database access, native TLS support, the syslog-ng 4 F2 now supports Application-levelAcknowledgement via Reliable Log Transport Protocol (RLTP)^TM , a new transport protocol that prevents message loss during connection breaks. In addition, the latest version of syslog-ng can now natively collect and process log messages from SQL databases enabling users to easily manage log messages from a wide variety of enterprise software and custom applications. syslog-ng 3.3.4 is released --------------------------- It is a bugfix release, which fixes all previously known problems in the 3.3 series. There is only one change in 3.3 sources since the last release: manual pages were put under the GPL, and XML sources are now also available, so that the entire source code of syslog-ng is free from this point onwards. Sources are available at http://www.balabit.com/network- <http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…>security/syslog-ng/opensource-logging-system/downloads/download <http://www.balabit.com/network-security/syslog-ng/opensource-logging-system…> Packages for some distributions are available from http://www.balabit.com/network-security/syslog-ng/opensource-logging-system… Detailed changelog is available at http://www.balabit.com/files/syslog-ng/open-source-edition/3.3.4/changelog-… Documentation was also updated: http://www.balabit.com/support/documentation/documents/syslog-ng-ose-3.3-gu… EU Data Protection Directive – How a single regulation could boost the transparency in IT security? -------------------------------- Personal opinion from Balázs Scheidler, CEO of BalaBit Overall, the *EU Data Protection Directive can be a milestone in boosting the transparency of IT security at* organizations – similarly to the regulatory compliance regulations after the Enron case. If adopted, the new directive could bring about a change in *the implementation of IT security policies so that the current focus on audits could shift to the deeper integration of IT security processes into business processes. *As logging and log management are the base of every monitoring method, *technologies with high-speed and zero message loss capabilities, like syslog-ng, will come to the front.* Encrypting log files, in which companies store user names, passwords and other sensitive company data is also key to prevent data loss. http://bscheidler.blogs.balabit.com/2012/02/eu-data-protection-directive-ho… OTHER SHORT NEWS * What makes the upcoming v3.4 even more flexible? It's summarized by Bazsi, lead developer of syslog-ng at http://bazsi.blogs.balabit.com/2012/01/syslog-ng-flexibility-improvements/ Watch out for an upcoming Alpha version! * There are still many syslog-ng v2.0 users. For those, here is a teaser, what changed since that release: http://czanik.blogs.balabit.com/2012/02/whats-new-in-syslog-ng-3-x/ * A syslog-ng presentation for FOSDEM: http://czanik.blogs.balabit.com/2012/02/fosdem-syslog-ng-as-upstream/ * ELSA, the high performance web GUI for syslog-ng, received a very useful new feature, post processing of search results: http://ossectools.blogspot.com/2012/01/hunting-with-elsa-transforms.html NEW RELEASES: * syslog-ng OSE 3.3.4: https://lists.balabit.hu/pipermail/syslog-ng-announce/2012-January/000133.h… * syslog-ng PE 4F2 ARCHIVE http://insider.blogs.balabit.com/ -- Peter Czanik (CzP)<czanik(a)balabit.hu> BalaBit IT Security / syslog-ng upstream http://czanik.blogs.balabit.com/

1 0

Extract portion of SDATA file name?
by Andika Daud 21 Feb '12

21 Feb '12

Hi, I'm a noobie and just about beginning to try out syslog-ng at my company. I have the following configuration on the client: source s_apache { file("/apps/log/apache/store-dev03/*" program_override("apache") flags(no-parse) ); }; destination d_remote { syslog("10.5.76.125" transport("tcp") port(514) # template("${MSGONLY}") ); }; log { source(s_apache); destination(d_remote); }; And something like this on the server side: source s_network { syslog(ip(0.0.0.0) port(514) transport("tcp") ); }; destination d_local { file("/apps/log/syslong-ng/$PROGRAM/$HOST/$YEAR$MONTH$DAY.log" # ignore the use of template below, I just wanted to prove the server side could # see the value of the filepath from the client template("${.SDATA.file(a)18372.4.name} - ${MSGONLY}\n") ); }; log { source(s_network); destination(d_local); }; My ("${.SDATA.file(a)18372.4.name} value can be something like this: '/apps/log/apache/store-dev03/access.log'. What I want to be able to do is to parse that path. That is to discard the begining of the path, /apps/log/apache. Get the rest of the value, especially the file name (access.log) to be used to construct destination path. Is this possible? Thank you for your kind help.

2 2