On Wed, 2010-09-15 at 09:51 -0500, Martin Holste wrote:
I guess syslog-ng could also write tab separated data into files and can also do per-minute batches (by using the $MIN macro). Are there any other things the perl stuff does?
Yes, you certainly could get Syslog-NG to write TSV in minute batches without any other program's intervention. I pipe to Perl because I do some minor data alterations before writing to file. Specifically, I CRC the program name to generate a program ID so that I can store the program in a programs table and the program_id in the main logs table to conserve space and keep the database in as close to 3rd normal form as I can.
Perfect example for a template-function.
I also do basic conversions like INET_ATON all IP addresses to store them in integer columns.
Again, should be a great example for a template-function. It was such a good idea, that I've actually coded it here: commit 70e91556b6af8724334443347fd6488745405344 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Mon Sep 20 17:12:27 2010 +0200 convertfuncs: new plugin to contain conversion template functions The plugin now only contains ipv4-to-int which converts an IPv4 address to a long integer. Usage: $(ipv4-to-int $SOURCEIP)
In the future, I would put advanced correlation capabilities there (probably nothing like what Matthew is cooking up, though!) as well as any real-time cluster messaging I need to do.
Expect a blog post on this topic, a simple correllation engine is now built into patterndb. I'm afraid the information about this topic is not very much, but anyway, here's the patch that implements it: commit 9d07e274bdf2ba00b0e697a13299140f4bf04ed3 Author: Balazs Scheidler <bazsi@balabit.hu> Date: Mon Sep 20 15:54:37 2010 +0200 db-parser: initial support for simple message correllation This feature is not yet complete, will probably leak memory, but if not leak will probably use a _lot_ of memory, but still makes it possible to use simple log event correllation for those who want it. -- Bazsi