[syslog-ng] advice/assistance with parsing attempt requested

[Bill Anderson]

On Dec 6, 2010, at 11:15 AM, Martin Holste wrote:

> Good points, Bill.
> 
> This is a cool challenge!

Aye, it sure is. :)

> 
> If the values can really come in any order and you don't necessary
> know all possible extra values ahead of time, then there's a good
> chance that regexp is your only hope, through Perl or other means.
> Pattern-db is really not setup to do this kind of thing, because the
> order changes.

Agreed, if the order is going to be fully dynamic I would write a Python script to generate the permutations as a patterndb file and go that route. ;) If that wasn't desired (or for some other reason didn't work), I'd probably go with a python daemon or c++ (I've got a lot of log parsing using Qt for example).

> 
> This must be pretty high volume, as I've got Perl doing regexp on
> around 3-4k large messages per second with no problems.  If that's the
> case, maybe you want a hybrid solution of some sort where you do some
> of the formatting in pattern-db, but then output to Perl for the final
> parsing and writing.

Agreed, Perl is plenty quick, hence my wondering about the actual volume. If it is too much for Perl I'd go w/C++.

> Logs -> Syslog-NG -> Perl master -> AIO to Perl Child n -> write file to disk

Personally, I'd make the last step routing back into syslog-ng with a source on a custom port and letting syslog handle the writing to disk. That way you can still use macros such as timestamps, etc.. Then again, that may be because I do that all the time. ;) A log statement that takes everything from the custom source and logs to a file should work beautifully; no need for filters though you could still do additional processing if needed. That said I'd also consider running a daemon that accepted all the input, formatted it, and then sent it to syslog-ng, pointing the clients at the custom daemon if that was possible.

One advantage to the daemon route is that it wouldn't *have* to reside on the same system.


Cheers,
Bill

--
Bill Anderson, RHCE
Linux Systems Engineer
bill.anderson at bodybuilding.com