[syslog-ng] Buffering AF_UNIX Destination, Batch Post Processing Messages

[Martin Holste]

> I have two cases I am trying to solve.
>
> 1) Crunch on the logs in 60 second batches to look for anomalies.
>
> For this case I will need:
>
> * all messages available in the memory of a single Perl
> process / thread / etc. to perform the computations
>
This should be no problem for a 60 second batch.  The technique was
borne from my attempt to have N child worker processes.  Instead of N,
I just have on child process.  This way, the Syslog-NG -> Perl parent
pipe stays open all the time, and Perl just swaps in a new child
process when the 60 second batch is up.  Oh, and use the Perl built-in
"alarm" command for that, as in:

while (1){ #main daemon loop
  my $fh;
  my $pid = open( $fh, "|-" );    # fork and send to child's STDIN
  if ($pid){
    #parent
    while (<>){
      $fh->print($_); # send logs to child worker
    }
  }
  else {
    #child
    my $continue = 1;
    local $SIG{ALRM} = sub { $continue = 0; }
    alarm 60;
    while ($continue and <>){ #this reads from the parent $fh->print()
      #do your log processing
    }
    #done with 60 second batch here, fork the anomaly cruncher and exit
  }
}

You will have to tweak this to do exactly what you want, probably with
a second fork, but that's a decent skeleton for how to chain processes
together without using anything too fancy.  Async frameworks like POE
and AnyEvent are a good fit for the fork management.

Incidentally, I'd be interested in seeing what you come up with for
the guts of the anomaly crunching, if you're willing to share.

--Martin