Hi all, I have a doubt about a process that i have. Maybe the solution is in the side of syslog-ng, i am searching. My process is: Log -> Perl Script -> Fifo -> Syslog-ng -------------- > Syslog-ng -> perl -> rrd The problem is when the application write to many lines in the fifo file (through the perl script), my syslog-ng (Standard Configuration) can't process all lines quickly, and the load of cpu rises a lot. Is there any parameters that i can use to optimize the reading? Someone have a similar solution? Thanks a lot. Att., Luiz
Hi, Luiz Cruz <luizcruz.msg@gmail.com> [20070604 09:43:08 -0200]:
Hi all,
I have a doubt about a process that i have. Maybe the solution is in the side of syslog-ng, i am searching. My process is:
Log -> Perl Script -> Fifo -> Syslog-ng -------------- > Syslog-ng -> perl -> rrd
Can't this be redone as: Perl Script ---(Sys::Syslog)---> syslog-ng -> syslog-ng -> Perl -> RRD You do not explain where and what 'Log' is so it is difficult to help out.
The problem is when the application write to many lines in the fifo file (through the perl script), my syslog-ng (Standard Configuration) can't process all lines quickly, and the load of cpu rises a lot. Is there any parameters that i can use to optimize the reading? Someone have a similar solution?
If 'Log' is a file on the machine why can you not just use the 'tail' functionality[1] of syslog-ng? Cheers Alex [1] http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch08s01.html#id...
Thanks a lot.
Att., Luiz _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu https://lists.balabit.hu/mailman/listinfo/syslog-ng Frequently asked questions at http://www.campin.net/syslog-ng/faq.html
Hi Alex, On 6/4/07, Alexander Clouter <ac56@soas.ac.uk> wrote:
Hi,
Luiz Cruz <luizcruz.msg@gmail.com> [20070604 09:43:08 -0200]:
Hi all,
I have a doubt about a process that i have. Maybe the solution is in the side of syslog-ng, i am searching. My process is:
Log -> Perl Script -> Fifo -> Syslog-ng -------------- > Syslog-ng -> perl -> rrd
Can't this be redone as:
Perl Script ---(Sys::Syslog)---> syslog-ng -> syslog-ng -> Perl -> RRD
You do not explain where and what 'Log' is so it is difficult to help out.
Nice, don´t use a fifo file sounds great. I will try. Log is from a jetty aplication. The request is proxyed by apache. But i have to parse the lines before sent to rrd (cacti). The log is about 1GB/day. Sorry if i didn´t mention before.
The problem is when the application write to many lines in the fifo file (through the perl script), my syslog-ng (Standard Configuration) can't process all lines quickly, and the load of cpu rises a lot. Is there any parameters that i can use to optimize the reading? Someone have a similar solution?
If 'Log' is a file on the machine why can you not just use the 'tail' functionality[1] of syslog-ng?
Cheers
Alex
[1] http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch08s01.html#id...
I didn´t find this... is that a pipe function?
Thanks a lot.
Att., Luiz
Again. Thanx! Luiz
Hi, Luiz Cruz <luizcruz.msg@gmail.com> [20070605 11:49:21 -0300]:
Nice, don?t use a fifo file sounds great. I will try. Log is from a jetty aplication. The request is proxyed by apache. But i have to parse the lines before sent to rrd (cacti). The log is about 1GB/day. Sorry if i didn?t mention before.
It was only because I could not tell if it was your Perl script directly generating the syslog messages or not. I was only curious why the pipe was there...looks like it was a case of "yay it works" :)
http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/ch08s01.html#id...
I didn?t find this... is that a pipe function?
If you are logging to a file and having Perl munch on a log file you can remove the perl script and pipe from the whole works. syslog-ng can be configured to read in directly a log file; it effectively does a 'tail -f' on the file. Cheers Alex
On 6/5/07, Alexander Clouter <ac56@soas.ac.uk> wrote:
If you are logging to a file and having Perl munch on a log file you can remove the perl script and pipe from the whole works. syslog-ng can be configured to read in directly a log file; it effectively does a 'tail -f' on the file.
Except for cases where you need the Perl script to post-process the log file to produce a normalized or custom syslog message format.
Hi, Tom Le <dottom@gmail.com> [20070605 21:31:33 -0700]:
On 6/5/07, Alexander Clouter <ac56@soas.ac.uk> wrote:
If you are logging to a file and having Perl munch on a log file you can remove the perl script and pipe from the whole works. syslog-ng can be configured to read in directly a log file; it effectively does a 'tail -f' on the file.
Except for cases where you need the Perl script to post-process the log file to produce a normalized or custom syslog message format.
Agreed, however a better design would be: <Log> ---> <Perl> ---> <Log2> <Log2> --[using 'tail' function]--> syslog-ng -> syslog-ng These would be disconnected and you would not run into any nasty problems. Another advantage being that if the perl code soaks up the CPU cycles and the logging is not urgent (needed at the final destination ASAP) you could nice down the priority of the process to prevent it stomping in on syslog-ng's performance. Cheers Alex
On 6/5/07, Alexander Clouter <ac56@soas.ac.uk> wrote:
If you are logging to a file and having Perl munch on a log file you
can
remove the perl script and pipe from the whole works. syslog-ng can be configured to read in directly a log file; it effectively does a 'tail -f' on the file.
Except for cases where you need the Perl script to post-process the log file to produce a normalized or custom syslog message format.
Agreed, however a better design would be:
<Log> ---> <Perl> ---> <Log2>
<Log2> --[using 'tail' function]--> syslog-ng -> syslog-ng
These would be disconnected and you would not run into any nasty problems. Another advantage being that if the perl code soaks up the CPU cycles and the logging is not urgent (needed at the final destination ASAP) you could nice down the priority of the process to prevent it stomping in on syslog-ng's performance.
But that scenario only supports logs that are "syslog friendly". If you have to do any kind of normalization or post-processing or dealing with multi-line records in a log file, you have to use a post-processing process such as Perl. Not all logs have human-friendly readable text and the post-processing Perl script can translate the log record into something more useful. The CPU cycles required by Perl parsing is minimal and is a common myth that Perl is a CPU hog. I have Perl scripts processing billions of log events per week.
Hi, Tom Le <dottom@gmail.com> [20070606 09:18:02 -0700]:
Agreed, however a better design would be:
<Log> ---> <Perl> ---> <Log2>
<Log2> --[using 'tail' function]--> syslog-ng -> syslog-ng
These would be disconnected and you would not run into any nasty problems. Another advantage being that if the perl code soaks up the CPU cycles and the logging is not urgent (needed at the final destination ASAP) you could nice down the priority of the process to prevent it stomping in on syslog-ng's performance.
But that scenario only supports logs that are "syslog friendly". If you have to do any kind of normalization or post-processing or dealing with multi-line records in a log file, you have to use a post-processing process such as Perl. Not all logs have human-friendly readable text and the post-processing Perl script can translate the log record into something more useful.
Well thats why the Perl step is still in there, to convert from the raw source to whatever is wanted.
The CPU cycles required by Perl parsing is minimal and is a common myth that Perl is a CPU hog. I have Perl scripts processing billions of log events per week.
Someone might want to do something CPU intensive but also it gives you the option of doing a stack of non-CPU bound activities that are DNS related and/or SQL related. Either way, there is no need for that pipe :) Cheers Alex
participants (3)
-
Alexander Clouter
-
Luiz Cruz
-
Tom Le