[syslog-ng] Buffering AF_UNIX Destination, Batch Post Processing Messages

[Matthew Hall]

On Wed, Sep 08, 2010 at 03:53:37PM -0500, Martin Holste wrote:
> This should be no problem for a 60 second batch. The technique was 
> borne from my attempt to have N child worker processes. Instead of N, 
> I just have on child process. This way, the Syslog-NG -> Perl parent 
> pipe stays open all the time, and Perl just swaps in a new child 
> process when the 60 second batch is up. Oh, and use the Perl built-in 
> "alarm" command for that, as in

Thanks for the code example. I was planning to use some kind of alarm 
signal based technique and it helps to see how one should implement 
this.

Forking off a new anomaly cruncher each time is problematic because I 
will be generating a lot of big data structures which need to be 
maintained across 12-24 hours in order to find the kind of anomalies I 
need to find. If I fork off new workers, then each worker forgets what 
has been stored in the structures previously.

If I am forking off workers I'd like a way for them to see the data 
structures without duplicating log messages or data structures due to 
the multiple buffer copies, context switches, etc. this will cause.

It was this exact part of the problem which motivated my mails. I have 
been unable to think of a good way to pull things out of a pipe, socket, 
etc. in 60 second batches in such a way that I could keep pulling into 
the new batch, while processing the last batch, without forgetting all 
of the statistics I had collected in between.

> Incidentally, I'd be interested in seeing what you come up with for
> the guts of the anomaly crunching, if you're willing to share.

Personally I have an open source full disclosure approach to security 
and I do not believe in hiding anything beyond the industry standard 30 
day warning window for vulnerabilities. However this tradition is 
unfortunately not shared by my employers who have been funding my work 
on anomaly detection.

We could however discuss this topic privately and decide if there could 
be a way to share some of my publicly available knowledge in a way that 
benefits your curiosity and the community at large.

> --Martin

Matthew.