<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#0050d0">
<font size="-1"><font face="Helvetica, Arial, sans-serif">Syslog-ng
already has the exact functionality you are looking for (at least as
far as I understand what youre wanting). Create a udp destination
driver, set flush_timeout to 60000 (60 seconds), and flush_lines to 0
(the default). Syslog-ng will queue all the destination messages until
the oldest message is 60 seconds old, and then flushes them all out at
once.<br>
<br>
<br>
</font></font><br>
Sent: Martedì 7 Settembre 2010 19.05.26<br>
From: Matthew Hall <a class="moz-txt-link-rfc2396E" href="mailto:mhall@mhcomputing.net"><mhall@mhcomputing.net></a><br>
To: <a class="moz-txt-link-abbreviated" href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a> <br>
Subject: [syslog-ng] Buffering AF_UNIX Destination, Batch Post
Processing Messages
<blockquote cite="mid:20100908010526.GA29965@mhcomputing.net"
type="cite">
<pre wrap="">Hello All,
I want to configure an AF_UNIX SOCK_DGRAM syslog-ng destination which
sends certain log messages to an external program for further processing
and analysis. This program should batch up the messages into 60 second
batches for processing.
Currently I am running into an architectural challenge in how I should
process the 60 second batch without slowing down the select which is
collecting the messages from the destination.
In the past when creating a similar kind of application in Java I
handled this by creating a huge dynamic array to store objects creating
from each incoming message, then passed the array reference to another
background thread for processing, and began building a new array in the
select thread.
Currently I am trying to solve this same basic problem in Perl, which
has poor threading support. I am investigating a few different options:
* use threads anyway-- not recommended by more expert Perl devs I asked
* prefork a process which listens to the AF_UNIX from syslog-ng, and
writes to some kind of buffered non blocking pipe with a really big
buffer-- not sure if such a pipe device actually exists, many pipes
block
* postfork a worker process which handles the 60 second batch-- problem
here is that you want to have a whole lot of long term state data which
is maintained between batches to help separate the needles from the
haystacks, and you could get weird behavior on the duplicated FDs that
are still being select()ed in the parent process which are copied into
the child.
* use some kind of message or job queue to copy things from a producer
process to a consumer process-- gearman, theschwartz, beanstalk,
rabbitmq, activemq, and poe::component::mq have been suggested-- this
would probably cause a lot of context switching and unwanted buffer copies
* see if there is something existing in syslog-ng that can help with
this situation. can it somehow be convinced to buffer things internally
for my process when my process is busy on a 60 second batch, or send in
60 second batch, etc. / whatever other clever people can dream up?
Is this a problem other people have dealt with before? What did you do
about this one? I want to get this right and avoid making a big mess or
reinventing the wheel.
Matthew.
______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
</body>
</html>