What backend database were you getting a single box to do 220k
inserts/sec sustained? The fastest I've ever seen is a little over
100k/sec with LOAD DATA INFILE in MySQL, though I haven't used
particularly beefy boxes. If your tablespace is RAM based, I guess I
could believe that, but that's a lot of RAM to allocate to long-term
log storage.
In my setups, I write to files out to disk (via a Perl program) and
then do an import of the data file, which is the fastest method I've
seen so far. What method are you using?
On Thu, Sep 2, 2010 at 7:01 PM, <syslogng@feystorm.net> wrote:
So, after months of work, we finally turned on our production
environment
for syslog collection. However, we hit one immediate snag. Currently
were
writing to the database, and the way the database works is that it
collects
enough data to fill a single block, and then it flushes out that block.
Well
every time it goes to flush the block out, the insert takes an extra
couple
milliseconds. Now when I'm doing about 220000 inserts a second, that
millisecond delay is significant. So basically syslog has to pause on
that
log statement while it waits for the database to flush. (1 out of 10
messages was getting dropped)
Now I tried to solve this by writing multiple destination drivers so
that a
second database thread could be processing while the first was flushing,
but
that didnt work as it appears syslog waits for the destination driver to
complete before it hands data off to the second driver.
Instead I managed to solve the problem by creating yet more syslog
processes. So basically the master process listens for data from all the
hosts. It then runs a match on the $PID and sends all even numbered PIDs
to
one syslog process, and all odd numbered PIDs to a second syslog
process.
This way both processes can be inserting to the database at the same
time.
It effectively cuts the amount of work each database thread does in
half, so
that when it has to pause to flush, it doesnt cause the syslog buffer to
fill up.
Ultimately my request is this, allow multiple destination drivers to
work at
the same time. I realize this is probably not a simple change, but seems
like it would be a significant speed enhancement.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation:
http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html