[syslog-ng] feature request (parallel processing)

[syslogng at feystorm.net]

No, we dont usually do that for hours at a time. Our normal rate is 
probably half that, but we'll frequently burst up to that for periods of 
5-20 min (were an email provider, these are incoming & outgoing email logs).

Sent: Thursday, September 02, 2010 7:21:07 PM
From: Martin Holste <mcholste at gmail.com>
To: Syslog-ng users' and developers' mailing list 
<syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] feature request (parallel processing)
> I'm on IBM Bladecenter (granted an older 4-core 2.2GHz) with EMC SAN
> on MySQL 5.1 with a large bulk_insert_buffer and bulk loading is only
> around 100k/sec, which is about what you get maximum of writing raw
> text to a filehandle (because that's pretty much what it's doing).
> You're saying Oracle 11g on one mid-range server will do a sustained
> 220k/sec for hours at a time?
>
> On Thu, Sep 2, 2010 at 8:04 PM,  <syslogng at feystorm.net> wrote:
>   
>> Enterprise level equipment. Oracle 11g on a HP dl360 backed by an EMC SAN
>> array over fiber channel.
>>
>> Sent: Thursday, September 02, 2010 7:00:41 PM
>> From: Clayton Dukes <cdukes at gmail.com>
>> To: Syslog-ng users' and developers' mailing list
>> <syslog-ng at lists.balabit.hu>
>> Subject: Re: [syslog-ng] feature request (parallel processing)
>>
>> Ditto,
>> I'd really like to know how you're getting that rate, please share :-)
>> ______________________________________________________________
>>
>> Clayton Dukes
>> ______________________________________________________________
>>
>>
>> On Thu, Sep 2, 2010 at 8:22 PM, Martin Holste <mcholste at gmail.com> wrote:
>>     
>>> What backend database were you getting a single box to do 220k
>>> inserts/sec sustained?  The fastest I've ever seen is a little over
>>> 100k/sec with LOAD DATA INFILE in MySQL, though I haven't used
>>> particularly beefy boxes.  If your tablespace is RAM based, I guess I
>>> could believe that, but that's a lot of RAM to allocate to long-term
>>> log storage.
>>>
>>> In my setups, I write to files out to disk (via a Perl program) and
>>> then do an import of the data file, which is the fastest method I've
>>> seen so far.  What method are you using?
>>>
>>> On Thu, Sep 2, 2010 at 7:01 PM,  <syslogng at feystorm.net> wrote:
>>>       
>>>> So, after months of work, we finally turned on our production
>>>> environment
>>>> for syslog collection. However, we hit one immediate snag. Currently
>>>> were
>>>> writing to the database, and the way the database works is that it
>>>> collects
>>>> enough data to fill a single block, and then it flushes out that block.
>>>> Well
>>>> every time it goes to flush the block out, the insert takes an extra
>>>> couple
>>>> milliseconds. Now when I'm doing about 220000 inserts a second, that
>>>> millisecond delay is significant. So basically syslog has to pause on
>>>> that
>>>> log statement while it waits for the database to flush. (1 out of 10
>>>> messages was getting dropped)
>>>>
>>>> Now I tried to solve this by writing multiple destination drivers so
>>>> that a
>>>> second database thread could be processing while the first was flushing,
>>>> but
>>>> that didnt work as it appears syslog waits for the destination driver to
>>>> complete before it hands data off to the second driver.
>>>>
>>>> Instead I managed to solve the problem by creating yet more syslog
>>>> processes. So basically the master process listens for data from all the
>>>> hosts. It then runs a match on the $PID and sends all even numbered PIDs
>>>> to
>>>> one syslog process, and all odd numbered PIDs to a second syslog
>>>> process.
>>>> This way both processes can be inserting to the database at the same
>>>> time.
>>>> It effectively cuts the amount of work each database thread does in
>>>> half, so
>>>> that when it has to pause to flush, it doesnt cause the syslog buffer to
>>>> fill up.
>>>>
>>>> Ultimately my request is this, allow multiple destination drivers to
>>>> work at
>>>> the same time. I realize this is probably not a simple change, but seems
>>>> like it would be a significant speed enhancement.
>>>>
>>>>
>>>>
>>>> ______________________________________________________________________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation:
>>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>>
>>>>
>>>>
>>>>         
>>> ______________________________________________________________________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation:
>>> http://www.balabit.com/support/documentation/?product=syslog-ng
>>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>>
>>>       
>> ________________________________
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>>
>>     
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100902/e75195ad/attachment-0001.htm