- syslog-ng - lists.balabit.hu

abandon libdbi and write a new sql driver?
by Patrick H. 22 Feb '11

22 Feb '11

Reading through the mailing list lately I've seen a fair amount of problems reported with libdbi. Given that it also doesnt support prepared statements which is a very important feature, I'm wondering if libdbi should just be dropped entirely. Libdbi seems to be of use to applications that want just very basic sql support, and not for anything involving high performance or reliability. I know its no small undertaking to write a new DBI for all the common databases out there, but if syslog-ng is supposed to be high-performing, libdbi does not fit in with that. I dont know if it'd be best to start a new project to replace libdbi but accomplish the same thing of providing a common API for all databases, or to just write separate modules for syslog-ng 3.2+ for each database out there. Writing modules would certainly be easier as they could be written independently by people who best know each database API. This is all just my opinion, but I for one am all for it. In my production environment, we do logging to a database, and libdbi just didnt have the flexibility or performance we needed, so I ended up writing a separate program which connects to the database to perform the inserts. Would be nice to avoid having to do that. I know I've also heard many recommendations of piping out to a perl script to do the inserts. A perl script should never do its job faster than C code. -Patrick

6 9

syslog-ng 3.2.2 OSE takes 100% CPU after source files get truncated.
by Mailing Lists 22 Feb '11

22 Feb '11

Hi, Not sure if syslog-ng has a bug tracker, but I couldn't find one anyway, so posting this here. Here is the problem: I have source file that is a log file that is been truncated periodically (by log rotate). When the truncation happens syslog-ng process starts taking 100% CPU, and not doing any logging. This behavior remains until source file receives a write. To reproduce: 1. configure syslog-ng with following lines: source s_logfiles { file("/var/log/applogs/mysql/mysqld.log" flags(no-parse) program_override("mysqld.log")); file("/var/log/applogs/mysql/mysql-slow.log" flags(no-parse) program_override("mysql-slow.log")); }; destination d_mysql_logall { file("/var/log/mysqlall.log"); }; log { source(s_logfiles); destination(d_mysql_logall);}; 2. start syslog-ng 3. output some text into source files: for i in /var/log/applogs/mysql/mysqld.log /var/log/applogs/mysql/mysql-slow.log ; do echo "some text" >> $i; done 4. truncate source files: for i in /var/log/applogs/mysql/mysqld.log /var/log/applogs/mysql/mysql-slow.log ; do > $i; done 5. observe syslog-ng takes 100% CPU, doing nothing. 6. output some text into source files again: for i in /var/log/applogs/mysql/mysqld.log /var/log/applogs/mysql/mysql-slow.log ; do echo "some text, again" >> $i; done 7. observe syslog-ng returned to normal operation. comment: some times you'll need going through steps 3 - 5 a few times to trigger the bug. My config: syslog-ng 3.2.2 Installer-Version: 3.2.2 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#1d3f396485eb47b1ff6aa18ac4f1c4cd51c0ea4c Compile-Date: Jan 20 2011 11:34:35 Enable-Threads: off Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: on Enable-SQL: off Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off Regards.

3 18

[Bug 109] New: building on Cygwin fails in modules/dbparser
by bugzilla＠bugzilla.balabit.com 22 Feb '11

22 Feb '11

https://bugzilla.balabit.com/show_bug.cgi?id=109 Summary: building on Cygwin fails in modules/dbparser Product: syslog-ng Version: 3.2.x Platform: PC OS/Version: Other Status: NEW Severity: blocker Priority: unspecified Component: syslog-ng AssignedTo: bazsi(a)balabit.hu ReportedBy: vinschen(a)redhat.com Type of the Report: regression Estimated Hours: 0.0 Created an attachment (id=35) --> (https://bugzilla.balabit.com/attachment.cgi?id=35) Patch to fix build problem in modules/dbparser I just tested the new implementation from git, and gcc fails to build libsyslog-ng-patterndb.dll.a since a lot of function calls are missing: libtool: link: gcc -std=gnu99 -shared .libs/radix.o .libs/patterndb.o .libs/t imerwheel.o .libs/patternize.o -lssl -lcrypto -lz -lrt /usr/lib/libgmodule-2.0.dll.a -L/usr/lib /usr/lib/libglib-2.0.dll.a /usr/lib/libintl.dll.a /usr/lib/libiconv.dll.a /usr/lib/libevtlog.dll.a -lresolv /usr/lib/libpcre.dll.a -O2 -Wl,--export-all-symbols -o .libs/cygsyslog-ng-patterndb-0.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libsyslog-ng-patterndb.dll.a Creating library file: .libs/libsyslog-ng-patterndb.dll.a .libs/radix.o: In function `r_new_pnode': modules/dbparser/radix.c:374: undefined reference to `_log_msg_get_value_handl e' modules/dbparser/radix.c:618: undefined reference to `_msg_limit_internal_mess age' modules/dbparser/radix.c:618: undefined reference to `_log_msg_get_value_name' modules/dbparser/radix.c:618: undefined reference to `_msg_event_create' modules/dbparser/radix.c:629: undefined reference to `_msg_limit_internal_mess age' modules/dbparser/radix.c:567: undefined reference to `_msg_event_create' modules/dbparser/radix.c:674: undefined reference to `_msg_limit_internal_mess age' modules/dbparser/radix.c:674: undefined reference to `_msg_event_create' etc. The reason is that the dependency to libsyslog is missing. Would you mind to apply the attached patch? It fixes the build problem and the resulting pdbtool appears to work fine. Thanks, Corinna -- Configure bugmail: https://bugzilla.balabit.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are watching all bug changes.

1 1

filter/parse based on certificate attributes?
by Christian Tramnitz 22 Feb '11

22 Feb '11

With syslog 3.0 and ssl support using mutual certificate authentication is it possible to parse/filter/tag based on attributes in the certificate? I'd like to have a common listener and then relay or append messages based on client-cert attribute values... (Sorry if this message is arriving more than once, I seem to have problems with posting through Gmane.) Regards, Christian

4 4

dbtype in SQL logging
by Hendrik Visage 22 Feb '11

22 Feb '11

Hi there, found the following entry in the Administrator's guide(s): <quote> columns Type: string list Default: "date", "facility", "level", "host", "program", "pid", "message" *Description:* Name of the columns storing the data in *fieldname [dbtype]*format. The *[dbtype]* parameter is optional, and specifies the type of the field. By default, syslog-ng creates *text* columns. Note that not every database engine can index text fields. </quote> So now before I go search the source code for answers, what types of "[dbtype]" is supported/handled by syslog-ng 3.2.2? More precisely, I'm running on HP-UX 11.31 and preference is using PostgreSQL Thank you HEndrik

2 1

syslog-ng 3.2.2
by Rakesh Rajasekharan 20 Feb '11

20 Feb '11

Hi , I was trying to compile the recently releases 3.2.2 version. I notoiced that the --enable-static-linking option is not present in the configure script. Can someone please confirm. Thanks, Rakesh

3 2

Default name value pair
by John Kristoff 20 Feb '11

20 Feb '11

Perhaps the functionality is there and I don't realize it, and this is not critical if it's not, but it would come in handy to be able to do something like the following: destination d_sql { sql( ... values( "${FOO:-default_value}", ...) In other words, being able to specify a default value for the name-value pair when one doesn't exist much like you can do in UNIX shell scripts. Similarly, but in my case otherwise, using patterndb, it would be handy to be able to set a name-value pair per <pattern>. I believe right now you can only do do based on the upper level <patterns> hierarchy. To help clarify the reasoning, I have a database table with a field that I'd like to fill with at least a default value, but that field might not exist in the log message in some cases. For instance, BIND 9.x query logs can log the "view", but unless you configure views for all the possible classes, which is generally not done, queries for other classes will not have the view identifier. Compare these common examples: <test_message>queries: info: client 127.0.0.1#1024: view lopback: query: 1.0.0.127.in-addr.arpa IN PTR +</test_message> <test_message>queries: info: client 192.0.2.1#1024: query: version.bind CH TXT + (192.0.2.1)</test_message> Right now the view field gets logged as a null, which is OK, but I'd rather it be inserted with a default view of 'default' in this example. John

2 1

duplicate config elements in sql destinations
by John Kristoff 20 Feb '11

20 Feb '11

I have a syslog-ng.conf that contains multiple destination sections that all begin like this: destination d_sql_[foo] { sql( type(pgsql) host("127.0.0.1") database("bar") ... "[foo]" differs between each section as well as the table name, columns and values. However, syslog-ng complains with the following: Internal error, duplicate configuration elements refer to the same persistent config; name='afsql_dd(pgsql,127.0.0.1,,bar)' Things appear to work, but apparently syslog-ng is confused by the duplicate database destination, even if the tables are different. Is this merely cosmetic or does this have wider implications? Perhaps they each end up sharing the same fifo buffer when they shouldn't for instance? John

2 1

[PATCH][patternize] support for custom delimiters
by Balint Kovacs 20 Feb '11

20 Feb '11

Hi, with some time at my hands over the weekend I was finally able finish support for multiple delimiters in 'pdbtool patternize'. Rationale: The current implementation of patternize only splits up log messages at spaces, which is inaccurate - while working with patterns, I found that most of the volatile content in messages is enclosed in or followed by some kind of separator character(s). So the best way of handling this would be to make the user be able to define the delimiters and provide a sane default. Implementation: The delimiter list is extracted from each message separately after tokenization (this way it delimiters do not affect frequent word collection) and then added to the end of the cluster key in the SLCT phase to ensure unicity. When printing the pattern, the delimiter list is popped from the cluster key and as it is in the same order as the extracted words, we print delimiter with the same index as the word's within the for loop. The user input needs to be sanitized, we have to make sure that the space char is always in the delimiter list and that there are no duplicates. I have also changed the the parser marker char from * to 0x1A ("substitute" char) to prevent parser "false positives" in case a words starts with *. I also added the class and provider properties to each rule to make the generated xml xsd-compliant. The default delimiter list is :&~?![]=,;()'" And the results: Running time and memory consumption have almost doubled(!), as far as I could test, this comes from the fact, that we have way many more tokens than with splitting at spaces, so with the current structure we can't really do much about that. BUT accuracy has increased quite a lot, I think there is a visible difference. For the log message: core.session(3): (svc/intra_HTTP:1486416/http): Server connection established; server_fd='46', server_address='AF_INET(1.2.3.4:80)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='AF_INET(1.2.3.4:52007)', server_protocol='TCP' With the tokenizing at spaces: core.session(3): @ESTRING:: @Server connection established; @ESTRING:: @@ESTRING:: @server_zone='Zone(internet, 0.0.0.0/0)', @ESTRING:: @server_protocol='TCP' And with the new default delimiters: core.session(3): (svc/intra_HTTP:@ESTRING::)@: Server connection established; server_fd='@ESTRING::'@, server_address='AF_INET(@ESTRING:::@80)', server_zone='Zone(internet, 0.0.0.0/0)', server_local='AF_INET(1.2.3.4:@ESTRING::)@', server_protocol='TCP' (please note that the log sample used for generating these patterns was grepped on intra_HTTP, that's why the zorp service name and server port 80 are found as a frequent words and not ESTRING'd) Regards, Balint

2 1

[development] open questions
by Balazs Scheidler 20 Feb '11

20 Feb '11

Hi, I had some distraction in the last two weeks, thus I was unable to follow the syslog-ng mailing list/emails as closely I would have wanted. I've finished reading all the unread emails in my syslog-ng folder, and I wanted to summarize the status quo: Bugfixes/maintenance (Bazsi): * I've merged 3.2 into 3.3 to get the latest changes, bugfixes. libsyslog-ng-patterndb.so (Corinna, Marius Tomaschewski, Peter Czanik): * causes problems on cygwin & SUSE, so probably an alternative solution is needed. * my current idea is either to use Corinna's idea and put pdbtool in /usr/lib/syslog-ng (which I personally don't really like, as /usr/lib is not the place for binaries), or that libsyslog-ng-patterndb.so should be moved /usr/lib _after_ compilation by the distribution build scripts. * Marius, Corinna what's your take on this issue? $(format-json) template function (Balint, Algernon, Matthew) * waiting for the generic value-pairs() implementation * it could probably be added to 3.2 as well, if value-pairs doesn't prove to be risky in itself (or else they'd have to be put in the tfjson module). mongodb dynamic values support (Algernon) * waiting for the value-pairs() implementation (Algernon) * this is 3.3 stuff systemd support (Marius Tomaschewski, Tom Gundersen, Algernon) * should be more-or-less ready, Algernon promised to rebase it against both 3.2 and 3.3 , so that I can integrate into both. LogQueue scalability (Bazsi): * I was working on making LogQueue more scalable -- Bazsi

2 3