Using a perl script with the pipe destination driver
I have a perl script that reads from a pipe using the following construct: while ( 1 ) { die "$pipe is not a pipe!" unless ( -p $pipe ); open PIPE, $pipe or die "Cannot open $pipe: $!"; my @lines = <PIPE>; close PIPE; for ( @lines ) { # do some stuff } } The "do some stuff part" loads info into a database (the messages are processed before being put into the database and two hashes keep cached information relevant to this in order to speed up the process). I can test using: cat syslog > pipe and the script does its stuff, loading single syslog messages or a days worth (around 3000 messages in 8 seconds). I then try to add a destination within syslog-ng.conf: destination d_pipe { pipe("/path/to/pipe"); }; and log { source(all); destination(d_pipe); }; This doesn't work as I expected. It seems as if I have some kind of locking problem; as I send test syslog messages nothing happens, then if I stop syslog-ng all the test messages I have added (using logger) are then processed by the script. I can then restart syslog-ng, send messages and again they are not read by the script until syslog-ng is stopped. I was able to use the construct: open PIPE, $pipe or die "Cannot open $pipe: $!"; while ( <LOGPIPE> ) { # do something e.g. test write to file }; as a test and this worked OK, seeming to process each message as I entered it (although I didn't try a volume test). Based on what I'd found so far I had a few questions: - can anyone explain what I think is the locking problem with the first approach (apologies if I'm asking a general unix or perl question, but testing the pipe outside of syslog-ng seemed to work ok)? - If I use the second approach are log messages written atomically such that I will get them one at a time? - One of the reasons I adopted the first approach is that I can throw a lot of messages at the pipe and still have them processed and put in the DB in a short space of time. I was concerned that if I used the second approach and syslog-ng was passing messages into the pipe faster than they could be processed, I would end up loosing messages. I'm sure that there is either an OS or syslog-ng buffer that is configurable but I assume its finite. We are logging a lot of events from network equipment and in the event of a "message storm" when network problems occur I wanted to ensure all of the messages are captured. I'm using perl 5.8 and syslog-ng 1.6.2 on Solaris 8. Jim Mozley
Jim Mozley <jim.mozley@exponential-e.com> - Tue, Apr 27, 2004:
my @lines = <PIPE>; for ( @lines ) { # do some stuff }
This causes reading until the End Of File of PIPE, then you process all lines. Instead, I suggest you try: while ( <PIPE> ) { # do some stuff } [ Syslog-NG opens a pipe to your program when it is started and closes it when it's stopped. It doesn't open a new pipe for each log line. ]
cat syslog > pipe
When cat finishes, it causes an EOF in the pipe (no program has the "pipe" file open for writing, so no more messages will come in), and your program starts processing. You should notice with a large syslog and a slow perl program that your processing starts when the cat finishes. -- Loïc Minier <lool@dooz.org>
Loic Minier wrote:
Jim Mozley <jim.mozley@exponential-e.com> - Tue, Apr 27, 2004:
my @lines = <PIPE>; for ( @lines ) { # do some stuff }
This causes reading until the End Of File of PIPE, then you process all lines.
Instead, I suggest you try:
while ( <PIPE> ) { # do some stuff }
[ Syslog-NG opens a pipe to your program when it is started and closes it when it's stopped. It doesn't open a new pipe for each log line. ]
Thanks, I understood this was the case for a pipe as in two commands with a | but didn't know it kept a fifo open too. Should have guessed this from the behaviour. Thanks, Jim Mozley
participants (2)
-
Jim Mozley
-
Loic Minier