On Dec 6, 2010, at 11:15 AM, Martin Holste wrote:
Good points, Bill.
This is a cool challenge!
Aye, it sure is. :)
If the values can really come in any order and you don't necessary know all possible extra values ahead of time, then there's a good chance that regexp is your only hope, through Perl or other means. Pattern-db is really not setup to do this kind of thing, because the order changes.
Agreed, if the order is going to be fully dynamic I would write a Python script to generate the permutations as a patterndb file and go that route. ;) If that wasn't desired (or for some other reason didn't work), I'd probably go with a python daemon or c++ (I've got a lot of log parsing using Qt for example).
This must be pretty high volume, as I've got Perl doing regexp on around 3-4k large messages per second with no problems. If that's the case, maybe you want a hybrid solution of some sort where you do some of the formatting in pattern-db, but then output to Perl for the final parsing and writing.
Agreed, Perl is plenty quick, hence my wondering about the actual volume. If it is too much for Perl I'd go w/C++.
Logs -> Syslog-NG -> Perl master -> AIO to Perl Child n -> write file to disk
Personally, I'd make the last step routing back into syslog-ng with a source on a custom port and letting syslog handle the writing to disk. That way you can still use macros such as timestamps, etc.. Then again, that may be because I do that all the time. ;) A log statement that takes everything from the custom source and logs to a file should work beautifully; no need for filters though you could still do additional processing if needed. That said I'd also consider running a daemon that accepted all the input, formatted it, and then sent it to syslog-ng, pointing the clients at the custom daemon if that was possible. One advantage to the daemon route is that it wouldn't *have* to reside on the same system. Cheers, Bill -- Bill Anderson, RHCE Linux Systems Engineer bill.anderson@bodybuilding.com