<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#0050d0">
<font size="-1"><font face="Helvetica, Arial, sans-serif">Enterprise
level equipment. Oracle 11g on a HP dl360 backed by an EMC SAN array
over fiber channel.<br>
</font></font><br>
Sent: Thursday, September 02, 2010 7:00:41 PM<br>
From: Clayton Dukes <a class="moz-txt-link-rfc2396E" href="mailto:cdukes@gmail.com"><cdukes@gmail.com></a><br>
To: Syslog-ng users' and developers' mailing list
<a class="moz-txt-link-rfc2396E" href="mailto:syslog-ng@lists.balabit.hu"><syslog-ng@lists.balabit.hu></a> <br>
Subject: Re: [syslog-ng] feature request (parallel processing)
<blockquote
cite="mid:AANLkTinm8_wM0B88+Rr2JXGTY0a4LQujf4LmMNm5aYLx@mail.gmail.com"
type="cite">Ditto,
<div>I'd really like to know how you're getting that rate, please
share :-)</div>
<div><br clear="all">
______________________________________________________________ <br>
<br>
Clayton Dukes<br>
______________________________________________________________<br>
<br>
<br>
<div class="gmail_quote">On Thu, Sep 2, 2010 at 8:22 PM, Martin
Holste <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">What
backend database were you getting a single box to do 220k<br>
inserts/sec sustained? The fastest I've ever seen is a little over<br>
100k/sec with LOAD DATA INFILE in MySQL, though I haven't used<br>
particularly beefy boxes. If your tablespace is RAM based, I guess I<br>
could believe that, but that's a lot of RAM to allocate to long-term<br>
log storage.<br>
<br>
In my setups, I write to files out to disk (via a Perl program) and<br>
then do an import of the data file, which is the fastest method I've<br>
seen so far. What method are you using?<br>
<div>
<div class="h5"><br>
On Thu, Sep 2, 2010 at 7:01 PM, <<a moz-do-not-send="true"
href="mailto:syslogng@feystorm.net">syslogng@feystorm.net</a>>
wrote:<br>
> So, after months of work, we finally turned on our production
environment<br>
> for syslog collection. However, we hit one immediate snag.
Currently were<br>
> writing to the database, and the way the database works is that it
collects<br>
> enough data to fill a single block, and then it flushes out that
block. Well<br>
> every time it goes to flush the block out, the insert takes an
extra couple<br>
> milliseconds. Now when I'm doing about 220000 inserts a second,
that<br>
> millisecond delay is significant. So basically syslog has to pause
on that<br>
> log statement while it waits for the database to flush. (1 out of
10<br>
> messages was getting dropped)<br>
><br>
> Now I tried to solve this by writing multiple destination drivers
so that a<br>
> second database thread could be processing while the first was
flushing, but<br>
> that didnt work as it appears syslog waits for the destination
driver to<br>
> complete before it hands data off to the second driver.<br>
><br>
> Instead I managed to solve the problem by creating yet more syslog<br>
> processes. So basically the master process listens for data from
all the<br>
> hosts. It then runs a match on the $PID and sends all even
numbered PIDs to<br>
> one syslog process, and all odd numbered PIDs to a second syslog
process.<br>
> This way both processes can be inserting to the database at the
same time.<br>
> It effectively cuts the amount of work each database thread does
in half, so<br>
> that when it has to pause to flush, it doesnt cause the syslog
buffer to<br>
> fill up.<br>
><br>
> Ultimately my request is this, allow multiple destination drivers
to work at<br>
> the same time. I realize this is probably not a simple change, but
seems<br>
> like it would be a significant speed enhancement.<br>
><br>
><br>
</div>
</div>
>
______________________________________________________________________________<br>
> Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation:<br>
> <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a moz-do-not-send="true"
href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
><br>
><br>
><br>
______________________________________________________________________________<br>
Member info: <a moz-do-not-send="true"
href="https://lists.balabit.hu/mailman/listinfo/syslog-ng"
target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a moz-do-not-send="true"
href="http://www.balabit.com/support/documentation/?product=syslog-ng"
target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a moz-do-not-send="true"
href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<pre wrap="">
<hr size="4" width="90%">
______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
</body>
</html>