On Wed, Sep 08, 2010 at 03:53:37PM -0500, Martin Holste wrote:
This should be no problem for a 60 second batch. The technique was borne from my attempt to have N child worker processes. Instead of N, I just have on child process. This way, the Syslog-NG -> Perl parent pipe stays open all the time, and Perl just swaps in a new child process when the 60 second batch is up. Oh, and use the Perl built-in "alarm" command for that, as in
Thanks for the code example. I was planning to use some kind of alarm signal based technique and it helps to see how one should implement this. Forking off a new anomaly cruncher each time is problematic because I will be generating a lot of big data structures which need to be maintained across 12-24 hours in order to find the kind of anomalies I need to find. If I fork off new workers, then each worker forgets what has been stored in the structures previously. If I am forking off workers I'd like a way for them to see the data structures without duplicating log messages or data structures due to the multiple buffer copies, context switches, etc. this will cause. It was this exact part of the problem which motivated my mails. I have been unable to think of a good way to pull things out of a pipe, socket, etc. in 60 second batches in such a way that I could keep pulling into the new batch, while processing the last batch, without forgetting all of the statistics I had collected in between.
Incidentally, I'd be interested in seeing what you come up with for the guts of the anomaly crunching, if you're willing to share.
Personally I have an open source full disclosure approach to security and I do not believe in hiding anything beyond the industry standard 30 day warning window for vulnerabilities. However this tradition is unfortunately not shared by my employers who have been funding my work on anomaly detection. We could however discuss this topic privately and decide if there could be a way to share some of my publicly available knowledge in a way that benefits your curiosity and the community at large.
--Martin
Matthew.