I have two cases I am trying to solve.
1) Crunch on the logs in 60 second batches to look for anomalies.
For this case I will need:
* all messages available in the memory of a single Perl process / thread / etc. to perform the computations
This should be no problem for a 60 second batch. The technique was borne from my attempt to have N child worker processes. Instead of N, I just have on child process. This way, the Syslog-NG -> Perl parent pipe stays open all the time, and Perl just swaps in a new child process when the 60 second batch is up. Oh, and use the Perl built-in "alarm" command for that, as in: while (1){ #main daemon loop my $fh; my $pid = open( $fh, "|-" ); # fork and send to child's STDIN if ($pid){ #parent while (<>){ $fh->print($_); # send logs to child worker } } else { #child my $continue = 1; local $SIG{ALRM} = sub { $continue = 0; } alarm 60; while ($continue and <>){ #this reads from the parent $fh->print() #do your log processing } #done with 60 second batch here, fork the anomaly cruncher and exit } } You will have to tweak this to do exactly what you want, probably with a second fork, but that's a decent skeleton for how to chain processes together without using anything too fancy. Async frameworks like POE and AnyEvent are a good fit for the fork management. Incidentally, I'd be interested in seeing what you come up with for the guts of the anomaly crunching, if you're willing to share. --Martin