[syslog-ng] feature request (parallel processing)

[syslogng at feystorm.net]

Enterprise level equipment. Oracle 11g on a HP dl360 backed by an EMC 
SAN array over fiber channel.

Sent: Thursday, September 02, 2010 7:00:41 PM
From: Clayton Dukes <cdukes at gmail.com>
To: Syslog-ng users' and developers' mailing list 
<syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] feature request (parallel processing)
> Ditto,
> I'd really like to know how you're getting that rate, please share :-)
>
> ______________________________________________________________
>
> Clayton Dukes
> ______________________________________________________________
>
>
> On Thu, Sep 2, 2010 at 8:22 PM, Martin Holste <mcholste at gmail.com 
> <mailto:mcholste at gmail.com>> wrote:
>
>     What backend database were you getting a single box to do 220k
>     inserts/sec sustained?  The fastest I've ever seen is a little over
>     100k/sec with LOAD DATA INFILE in MySQL, though I haven't used
>     particularly beefy boxes.  If your tablespace is RAM based, I guess I
>     could believe that, but that's a lot of RAM to allocate to long-term
>     log storage.
>
>     In my setups, I write to files out to disk (via a Perl program) and
>     then do an import of the data file, which is the fastest method I've
>     seen so far.  What method are you using?
>
>     On Thu, Sep 2, 2010 at 7:01 PM,  <syslogng at feystorm.net
>     <mailto:syslogng at feystorm.net>> wrote:
>     > So, after months of work, we finally turned on our production
>     environment
>     > for syslog collection. However, we hit one immediate snag.
>     Currently were
>     > writing to the database, and the way the database works is that
>     it collects
>     > enough data to fill a single block, and then it flushes out that
>     block. Well
>     > every time it goes to flush the block out, the insert takes an
>     extra couple
>     > milliseconds. Now when I'm doing about 220000 inserts a second, that
>     > millisecond delay is significant. So basically syslog has to
>     pause on that
>     > log statement while it waits for the database to flush. (1 out of 10
>     > messages was getting dropped)
>     >
>     > Now I tried to solve this by writing multiple destination
>     drivers so that a
>     > second database thread could be processing while the first was
>     flushing, but
>     > that didnt work as it appears syslog waits for the destination
>     driver to
>     > complete before it hands data off to the second driver.
>     >
>     > Instead I managed to solve the problem by creating yet more syslog
>     > processes. So basically the master process listens for data from
>     all the
>     > hosts. It then runs a match on the $PID and sends all even
>     numbered PIDs to
>     > one syslog process, and all odd numbered PIDs to a second syslog
>     process.
>     > This way both processes can be inserting to the database at the
>     same time.
>     > It effectively cuts the amount of work each database thread does
>     in half, so
>     > that when it has to pause to flush, it doesnt cause the syslog
>     buffer to
>     > fill up.
>     >
>     > Ultimately my request is this, allow multiple destination
>     drivers to work at
>     > the same time. I realize this is probably not a simple change,
>     but seems
>     > like it would be a significant speed enhancement.
>     >
>     >
>     >
>     ______________________________________________________________________________
>     > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     > Documentation:
>     > http://www.balabit.com/support/documentation/?product=syslog-ng
>     > FAQ: http://www.campin.net/syslog-ng/faq.html
>     >
>     >
>     >
>     ______________________________________________________________________________
>     Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>     Documentation:
>     http://www.balabit.com/support/documentation/?product=syslog-ng
>     FAQ: http://www.campin.net/syslog-ng/faq.html
>
>
> ------------------------------------------------------------------------
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100902/a9d7d4c3/attachment.htm