February 2011 - syslog-ng - lists.balabit.hu

Default name value pair
by John Kristoff 20 Feb '11

20 Feb '11

Perhaps the functionality is there and I don't realize it, and this is not critical if it's not, but it would come in handy to be able to do something like the following: destination d_sql { sql( ... values( "${FOO:-default_value}", ...) In other words, being able to specify a default value for the name-value pair when one doesn't exist much like you can do in UNIX shell scripts. Similarly, but in my case otherwise, using patterndb, it would be handy to be able to set a name-value pair per <pattern>. I believe right now you can only do do based on the upper level <patterns> hierarchy. To help clarify the reasoning, I have a database table with a field that I'd like to fill with at least a default value, but that field might not exist in the log message in some cases. For instance, BIND 9.x query logs can log the "view", but unless you configure views for all the possible classes, which is generally not done, queries for other classes will not have the view identifier. Compare these common examples: <test_message>queries: info: client 127.0.0.1#1024: view lopback: query: 1.0.0.127.in-addr.arpa IN PTR +</test_message> <test_message>queries: info: client 192.0.2.1#1024: query: version.bind CH TXT + (192.0.2.1)</test_message> Right now the view field gets logged as a null, which is OK, but I'd rather it be inserted with a default view of 'default' in this example. John

2 1

duplicate config elements in sql destinations
by John Kristoff 20 Feb '11

20 Feb '11

I have a syslog-ng.conf that contains multiple destination sections that all begin like this: destination d_sql_[foo] { sql( type(pgsql) host("127.0.0.1") database("bar") ... "[foo]" differs between each section as well as the table name, columns and values. However, syslog-ng complains with the following: Internal error, duplicate configuration elements refer to the same persistent config; name='afsql_dd(pgsql,127.0.0.1,,bar)' Things appear to work, but apparently syslog-ng is confused by the duplicate database destination, even if the tables are different. Is this merely cosmetic or does this have wider implications? Perhaps they each end up sharing the same fifo buffer when they shouldn't for instance? John

2 1

[PATCH][patternize] support for custom delimiters
by Balint Kovacs 20 Feb '11

20 Feb '11

Hi, with some time at my hands over the weekend I was finally able finish support for multiple delimiters in 'pdbtool patternize'. Rationale: The current implementation of patternize only splits up log messages at spaces, which is inaccurate - while working with patterns, I found that most of the volatile content in messages is enclosed in or followed by some kind of separator character(s). So the best way of handling this would be to make the user be able to define the delimiters and provide a sane default. Implementation: The delimiter list is extracted from each message separately after tokenization (this way it delimiters do not affect frequent word collection) and then added to the end of the cluster key in the SLCT phase to ensure unicity. When printing the pattern, the delimiter list is popped from the cluster key and as it is in the same order as the extracted words, we print delimiter with the same index as the word's within the for loop. The user input needs to be sanitized, we have to make sure that the space char is always in the delimiter list and that there are no duplicates. I have also changed the the parser marker char from * to 0x1A ("substitute" char) to prevent parser "false positives" in case a words starts with *. I also added the class and provider properties to each rule to make the generated xml xsd-compliant. The default delimiter list is :&~?![]=,;()'" And the results: Running time and memory consumption have almost doubled(!), as far as I could test, this comes from the fact, that we have way many more tokens than with splitting at spaces, so with the current structure we can't really do much about that. BUT accuracy has increased quite a lot, I think there is a visible difference. For the log message: core.session(3): (svc/intra_HTTP:1486416/http): Server connection established; server_fd='46', server_address='AF_INET(1.2.3.4:80)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='AF_INET(1.2.3.4:52007)', server_protocol='TCP' With the tokenizing at spaces: core.session(3): @ESTRING:: @Server connection established; @ESTRING:: @@ESTRING:: @server_zone='Zone(internet, 0.0.0.0/0)', @ESTRING:: @server_protocol='TCP' And with the new default delimiters: core.session(3): (svc/intra_HTTP:@ESTRING::)@: Server connection established; server_fd='@ESTRING::'@, server_address='AF_INET(@ESTRING:::@80)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='AF_INET(1.2.3.4:@ESTRING::)@', server_protocol='TCP' (please note that the log sample used for generating these patterns was grepped on intra_HTTP, that's why the zorp service name and server port 80 are found as a frequent words and not ESTRING'd) Regards, Balint

2 1

[development] open questions
by Balazs Scheidler 20 Feb '11

20 Feb '11

Hi, I had some distraction in the last two weeks, thus I was unable to follow the syslog-ng mailing list/emails as closely I would have wanted. I've finished reading all the unread emails in my syslog-ng folder, and I wanted to summarize the status quo: Bugfixes/maintenance (Bazsi): * I've merged 3.2 into 3.3 to get the latest changes, bugfixes. libsyslog-ng-patterndb.so (Corinna, Marius Tomaschewski, Peter Czanik): * causes problems on cygwin & SUSE, so probably an alternative solution is needed. * my current idea is either to use Corinna's idea and put pdbtool in /usr/lib/syslog-ng (which I personally don't really like, as /usr/lib is not the place for binaries), or that libsyslog-ng-patterndb.so should be moved /usr/lib _after_ compilation by the distribution build scripts. * Marius, Corinna what's your take on this issue? $(format-json) template function (Balint, Algernon, Matthew) * waiting for the generic value-pairs() implementation * it could probably be added to 3.2 as well, if value-pairs doesn't prove to be risky in itself (or else they'd have to be put in the tfjson module). mongodb dynamic values support (Algernon) * waiting for the value-pairs() implementation (Algernon) * this is 3.3 stuff systemd support (Marius Tomaschewski, Tom Gundersen, Algernon) * should be more-or-less ready, Algernon promised to rebase it against both 3.2 and 3.3 , so that I can integrate into both. LogQueue scalability (Bazsi): * I was working on making LogQueue more scalable -- Bazsi

2 3

syslog-ng 3.01 compilation error on Solaris
by Rakesh Rajasekharan 20 Feb '11

20 Feb '11

Hi, I was trying to compile the syslog-ng 3.01 on Solaris 10 Opteron machine using the mixed linking option. In the configure stage I get the below error complaining about the glib . I have compiled glib with teh enable static flag and the libglib-2.0.a library is available under the path /usr/local/lib and my PKG_CONFIG_PATH environment variable has been set correctly too. Please help me solve this. * * Below is the configure error. checking pthread.h usability... yes checking pthread.h presence... yes checking for pthread.h... yes checking for pthread_create in -lpthread... yes checking for LIBNET... no checking whether to enable Sun STREAMS support... yes checking whether to enable Sun door support... yes checking whether to enable TCP wrapper support... no checking whether to enable spoof source support... no checking whether to enable IPv6 support... yes checking whether to enable SQL support... no checking whether to enable Linux capability support... no checking whether to enable PCRE support... no checking for pkg-config... (cached) /usr/local/bin/pkg-config checking pkg-config is at least version 0.14... yes checking for GLIB - version >= 2.10.1... yes (version 2.16.5) checking for EVTLOG... yes checking for static GLib libraries... no checking sanity checking Glib headers... yes *configure: error: static GLib libraries not found (a file named libglib-2.0.a), either link GLib dynamically using the --enable-dynamic-linking or install a static GLib* * * * Thanks & Regards, Rakesh *

2 4

Invitation to connect on LinkedIn
by Janez Barbic 19 Feb '11

19 Feb '11

LinkedIn ------------ Syslog-ng, I'd like to add you to my professional network on LinkedIn. - Janez Janez Barbic Student at Univerza v Ljubljani Slovenia Confirm that you know Janez Barbic https://www.linkedin.com/e/-uz7o7o-gkcu3i8u-2k/isd/2362025712/_2EaqCuR/ -- (c) 2011, LinkedIn Corporation

1 0

patterndb project?
by Valentijn Sessink 18 Feb '11

18 Feb '11

Hi, http://www.balabit.com/wiki/patterndb mentions the "patterndb project", but there is hardly any information. I have a couple of additions to the sshd patterns, does anyone know where to put/send them? Best regards, Valentijn

2 3

[Bug 112] New: Problems when using encoding in 3.1.3 ose edition
by bugzilla＠bugzilla.balabit.com 17 Feb '11

17 Feb '11

https://bugzilla.balabit.com/show_bug.cgi?id=112 Summary: Problems when using encoding in 3.1.3 ose edition Product: syslog-ng Version: 3.1.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: aseques(a)gmail.com Type of the Report: --- Estimated Hours: 0.0 Good morning, I needed to reencode some windows logs emited by snare so they are stored ad UTF on the fly. After reading the documentation, I found what seems to be the solution. Instead of creating a single source of data. I create an extra one, called src_win. source src { udp(ip("192.168.1.200") port(514)); }; source src_win { udp(ip("192.168.1.200") port(514) encoding("WINDOWS-1252")); }; Whenever I want to use the filter encoding for the logs, I will use this source instead of the generic one. log { source(src_win); filter(f_host_name); filter(f_messages); destination(name_messages); }; The problem is that as soon as I enable the source src_win for any log, it just stop logging the rest of sources. Is that a known bug or something I should fix. -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 3

patterndb single byte optional
by Jean F. Mousinho 17 Feb '11

17 Feb '11

Hello, I was wondering if there's any way of reducing two lines of pattern, which provide a way of matching when a optional byte is present or not, into a single line. Example: JAN @STRING:.day@ 2011 JAN @STRING:.day@ 2011 Notice that this is the case when day doesn't have the leading 0 and I don't want the .day to contain any space. Just asking, this would reduce my patterns to half :) but I can live with that... Maybe a new type of STRING, like @DSTRING:.day: @ (dirty string, with dirty chars inside the optional argument) would fix it? Thanks for your time. Jean-François Mousinho

2 1

Possible destination driver question
by Bill Anderson 15 Feb '11

15 Feb '11

I've had it in my "backlog" to write a couple destinations: redis and AMQP (specifically at the moment rabbitmq). I've done the Proof-of-Concept via program() already and of course it works. With all the recent announcements of people writing to various destinations such as SMTP and MongoDB, I thought I'd throw out an email to see if someone is already working on one before getting started on one sometime in the next couple sprints/weeks. Cheers, Bill -- Bill Anderson, RHCE Linux Systems Engineer bill.anderson(a)bodybuilding.com

2 1