Hi, Tom Le <dottom@gmail.com> [20070606 09:18:02 -0700]:
Agreed, however a better design would be:
<Log> ---> <Perl> ---> <Log2>
<Log2> --[using 'tail' function]--> syslog-ng -> syslog-ng
These would be disconnected and you would not run into any nasty problems. Another advantage being that if the perl code soaks up the CPU cycles and the logging is not urgent (needed at the final destination ASAP) you could nice down the priority of the process to prevent it stomping in on syslog-ng's performance.
But that scenario only supports logs that are "syslog friendly". If you have to do any kind of normalization or post-processing or dealing with multi-line records in a log file, you have to use a post-processing process such as Perl. Not all logs have human-friendly readable text and the post-processing Perl script can translate the log record into something more useful.
Well thats why the Perl step is still in there, to convert from the raw source to whatever is wanted.
The CPU cycles required by Perl parsing is minimal and is a common myth that Perl is a CPU hog. I have Perl scripts processing billions of log events per week.
Someone might want to do something CPU intensive but also it gives you the option of doing a stack of non-CPU bound activities that are DNS related and/or SQL related. Either way, there is no need for that pipe :) Cheers Alex