Ditto,<div>I'd really like to know how you're getting that rate, please share :-)</div><div><br clear="all">______________________________________________________________ <br><br>Clayton Dukes<br>______________________________________________________________<br>
<br><br><div class="gmail_quote">On Thu, Sep 2, 2010 at 8:22 PM, Martin Holste <span dir="ltr"><<a href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
What backend database were you getting a single box to do 220k<br>
inserts/sec sustained? The fastest I've ever seen is a little over<br>
100k/sec with LOAD DATA INFILE in MySQL, though I haven't used<br>
particularly beefy boxes. If your tablespace is RAM based, I guess I<br>
could believe that, but that's a lot of RAM to allocate to long-term<br>
log storage.<br>
<br>
In my setups, I write to files out to disk (via a Perl program) and<br>
then do an import of the data file, which is the fastest method I've<br>
seen so far. What method are you using?<br>
<div><div></div><div class="h5"><br>
On Thu, Sep 2, 2010 at 7:01 PM, <<a href="mailto:syslogng@feystorm.net">syslogng@feystorm.net</a>> wrote:<br>
> So, after months of work, we finally turned on our production environment<br>
> for syslog collection. However, we hit one immediate snag. Currently were<br>
> writing to the database, and the way the database works is that it collects<br>
> enough data to fill a single block, and then it flushes out that block. Well<br>
> every time it goes to flush the block out, the insert takes an extra couple<br>
> milliseconds. Now when I'm doing about 220000 inserts a second, that<br>
> millisecond delay is significant. So basically syslog has to pause on that<br>
> log statement while it waits for the database to flush. (1 out of 10<br>
> messages was getting dropped)<br>
><br>
> Now I tried to solve this by writing multiple destination drivers so that a<br>
> second database thread could be processing while the first was flushing, but<br>
> that didnt work as it appears syslog waits for the destination driver to<br>
> complete before it hands data off to the second driver.<br>
><br>
> Instead I managed to solve the problem by creating yet more syslog<br>
> processes. So basically the master process listens for data from all the<br>
> hosts. It then runs a match on the $PID and sends all even numbered PIDs to<br>
> one syslog process, and all odd numbered PIDs to a second syslog process.<br>
> This way both processes can be inserting to the database at the same time.<br>
> It effectively cuts the amount of work each database thread does in half, so<br>
> that when it has to pause to flush, it doesnt cause the syslog buffer to<br>
> fill up.<br>
><br>
> Ultimately my request is this, allow multiple destination drivers to work at<br>
> the same time. I realize this is probably not a simple change, but seems<br>
> like it would be a significant speed enhancement.<br>
><br>
><br>
</div></div>> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation:<br>
> <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
><br>
><br>
><br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</blockquote></div><br></div>