[syslog-ng] syslog-ng with sqlite 100% cpu

[work ran]

i am not sure if you mean about transaction with batch messages but i saw
that the older syslog-ng doest support it and that there was a fix so it
will support. I added the fix to the libdbi_drivers SRPM and created a new
RPM. the transaction work now. i can see the change. before my fix it
couldnt handle more them 100 messages per second. now it can handle more
then 700 per second.


On Fri, Feb 17, 2017 at 3:59 AM, Balazs Scheidler <bazsi77 at gmail.com> wrote:

> The problem is that older syslog-ng versions didn't have transaction
> support, so even if you set flush-lines() higher, it may not be starting
> transactions.
>
> On Tue, Feb 14, 2017 at 11:57 PM, work ran <work.greeran at gmail.com> wrote:
>
>> thanks for the response
>> For now our architecture is limiting us to an old syslog-ng version so we
>> cannot upgrade.
>> as for the flush we are running it with 200 and with timeout 200 also. we
>> are trying to increase the flush-lines to 1000 and see if it helps.
>> i also googled the problem and saw a couple of mails that said that there
>> was a bug with syslog-ng 3.2.5/4 and pcre that created a bug with the
>> cpu usage. does someone if it might be the problem and how to fix it.
>>
>> thanks
>> Ran
>>
>>
>> On Tue, Feb 14, 2017 at 5:53 PM, Scheidler, Balázs <
>> balazs.scheidler at balabit.com> wrote:
>>
>>> It might be because of transaction size. Sqlite is very inefficient if
>>> the transaction size is small.
>>>
>>> Newer syslog-ng versions support batching, set flush-lines to something
>>> like 100 and it would improve a lot.
>>>
>>> On Feb 14, 2017 06:31, "Fekete, Róbert" <robert.fekete at balabit.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> It is not normal, it might be a bug. However, version 3.2 is ancient,
>>>> I'd recommend upgrading to a newer version. You can find release tarballs
>>>> on github, or links to binaries here: https://syslog-ng.org/3r
>>>> d-party-binaries/
>>>>
>>>> Regards,
>>>>
>>>> Robert
>>>>
>>>> On Tue, Feb 14, 2017 at 3:19 PM, work ran <work.greeran at gmail.com>
>>>> wrote:
>>>>
>>>>> hi
>>>>> i am running syslog-ng with sqlite
>>>>> i am sending 50 logs per second and i am getting 80% cpu usage.
>>>>> could someone tell me how i can fix this or is it normal
>>>>> thanks
>>>>> Ran
>>>>>
>>>>>
>>>>>
>>>>> syslog-ng 3.2.5
>>>>> Installer-Version: 3.2.5
>>>>> Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng
>>>>> -ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116
>>>>> Compile-Date: Jul 25 2014 15:20:50
>>>>> Enable-Threads: on
>>>>> Enable-Debug: off
>>>>> Enable-GProf: off
>>>>> Enable-Memtrace: off
>>>>> Enable-Sun-STREAMS: off
>>>>> Enable-IPv6: on
>>>>> Enable-Spoof-Source: on
>>>>> Enable-TCP-Wrapper: on
>>>>> Enable-SSL: off
>>>>> Enable-SQL: on
>>>>> Enable-Linux-Caps: off
>>>>> Enable-Pcre: on
>>>>> Enable-Pacct: off
>>>>>
>>>>>
>>>>>
>>>>> ____________________________________________________________
>>>>> __________________
>>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> Documentation: http://www.balabit.com/support
>>>>> /documentation/?product=syslog-ng
>>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>>
>>>>>
>>>>>
>>>>
>>>> ____________________________________________________________
>>>> __________________
>>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> Documentation: http://www.balabit.com/support
>>>> /documentation/?product=syslog-ng
>>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>>
>>>>
>>>>
>>> ____________________________________________________________
>>> __________________
>>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> Documentation: http://www.balabit.com/support
>>> /documentation/?product=syslog-ng
>>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>>
>>>
>>>
>>
>> ____________________________________________________________
>> __________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=
>> syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
>>
>
>
> --
> Bazsi
>
> ____________________________________________________________
> __________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?
> product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20170219/5264f8d1/attachment.html>