syslog-ng with sqlite 100% cpu
hi i am running syslog-ng with sqlite i am sending 50 logs per second and i am getting 80% cpu usage. could someone tell me how i can fix this or is it normal thanks Ran syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit //var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jul 25 2014 15:20:50 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off
Hi, It is not normal, it might be a bug. However, version 3.2 is ancient, I'd recommend upgrading to a newer version. You can find release tarballs on github, or links to binaries here: https://syslog-ng.org/3rd-party-binaries/ Regards, Robert On Tue, Feb 14, 2017 at 3:19 PM, work ran <work.greeran@gmail.com> wrote:
hi i am running syslog-ng with sqlite i am sending 50 logs per second and i am getting 80% cpu usage. could someone tell me how i can fix this or is it normal thanks Ran
syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog- ng-ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jul 25 2014 15:20:50 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
It might be because of transaction size. Sqlite is very inefficient if the transaction size is small. Newer syslog-ng versions support batching, set flush-lines to something like 100 and it would improve a lot. On Feb 14, 2017 06:31, "Fekete, Róbert" <robert.fekete@balabit.com> wrote:
Hi,
It is not normal, it might be a bug. However, version 3.2 is ancient, I'd recommend upgrading to a newer version. You can find release tarballs on github, or links to binaries here: https://syslog-ng.org/ 3rd-party-binaries/
Regards,
Robert
On Tue, Feb 14, 2017 at 3:19 PM, work ran <work.greeran@gmail.com> wrote:
hi i am running syslog-ng with sqlite i am sending 50 logs per second and i am getting 80% cpu usage. could someone tell me how i can fix this or is it normal thanks Ran
syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng -ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jul 25 2014 15:20:50 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
thanks for the response For now our architecture is limiting us to an old syslog-ng version so we cannot upgrade. as for the flush we are running it with 200 and with timeout 200 also. we are trying to increase the flush-lines to 1000 and see if it helps. i also googled the problem and saw a couple of mails that said that there was a bug with syslog-ng 3.2.5/4 and pcre that created a bug with the cpu usage. does someone if it might be the problem and how to fix it. thanks Ran On Tue, Feb 14, 2017 at 5:53 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
It might be because of transaction size. Sqlite is very inefficient if the transaction size is small.
Newer syslog-ng versions support batching, set flush-lines to something like 100 and it would improve a lot.
On Feb 14, 2017 06:31, "Fekete, Róbert" <robert.fekete@balabit.com> wrote:
Hi,
It is not normal, it might be a bug. However, version 3.2 is ancient, I'd recommend upgrading to a newer version. You can find release tarballs on github, or links to binaries here: https://syslog-ng.org/3r d-party-binaries/
Regards,
Robert
On Tue, Feb 14, 2017 at 3:19 PM, work ran <work.greeran@gmail.com> wrote:
hi i am running syslog-ng with sqlite i am sending 50 logs per second and i am getting 80% cpu usage. could someone tell me how i can fix this or is it normal thanks Ran
syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng -ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jul 25 2014 15:20:50 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
The problem is that older syslog-ng versions didn't have transaction support, so even if you set flush-lines() higher, it may not be starting transactions. On Tue, Feb 14, 2017 at 11:57 PM, work ran <work.greeran@gmail.com> wrote:
thanks for the response For now our architecture is limiting us to an old syslog-ng version so we cannot upgrade. as for the flush we are running it with 200 and with timeout 200 also. we are trying to increase the flush-lines to 1000 and see if it helps. i also googled the problem and saw a couple of mails that said that there was a bug with syslog-ng 3.2.5/4 and pcre that created a bug with the cpu usage. does someone if it might be the problem and how to fix it.
thanks Ran
On Tue, Feb 14, 2017 at 5:53 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
It might be because of transaction size. Sqlite is very inefficient if the transaction size is small.
Newer syslog-ng versions support batching, set flush-lines to something like 100 and it would improve a lot.
On Feb 14, 2017 06:31, "Fekete, Róbert" <robert.fekete@balabit.com> wrote:
Hi,
It is not normal, it might be a bug. However, version 3.2 is ancient, I'd recommend upgrading to a newer version. You can find release tarballs on github, or links to binaries here: https://syslog-ng.org/3r d-party-binaries/
Regards,
Robert
On Tue, Feb 14, 2017 at 3:19 PM, work ran <work.greeran@gmail.com> wrote:
hi i am running syslog-ng with sqlite i am sending 50 logs per second and i am getting 80% cpu usage. could someone tell me how i can fix this or is it normal thanks Ran
syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng -ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jul 25 2014 15:20:50 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
i am not sure if you mean about transaction with batch messages but i saw that the older syslog-ng doest support it and that there was a fix so it will support. I added the fix to the libdbi_drivers SRPM and created a new RPM. the transaction work now. i can see the change. before my fix it couldnt handle more them 100 messages per second. now it can handle more then 700 per second. On Fri, Feb 17, 2017 at 3:59 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
The problem is that older syslog-ng versions didn't have transaction support, so even if you set flush-lines() higher, it may not be starting transactions.
On Tue, Feb 14, 2017 at 11:57 PM, work ran <work.greeran@gmail.com> wrote:
thanks for the response For now our architecture is limiting us to an old syslog-ng version so we cannot upgrade. as for the flush we are running it with 200 and with timeout 200 also. we are trying to increase the flush-lines to 1000 and see if it helps. i also googled the problem and saw a couple of mails that said that there was a bug with syslog-ng 3.2.5/4 and pcre that created a bug with the cpu usage. does someone if it might be the problem and how to fix it.
thanks Ran
On Tue, Feb 14, 2017 at 5:53 PM, Scheidler, Balázs < balazs.scheidler@balabit.com> wrote:
It might be because of transaction size. Sqlite is very inefficient if the transaction size is small.
Newer syslog-ng versions support batching, set flush-lines to something like 100 and it would improve a lot.
On Feb 14, 2017 06:31, "Fekete, Róbert" <robert.fekete@balabit.com> wrote:
Hi,
It is not normal, it might be a bug. However, version 3.2 is ancient, I'd recommend upgrading to a newer version. You can find release tarballs on github, or links to binaries here: https://syslog-ng.org/3r d-party-binaries/
Regards,
Robert
On Tue, Feb 14, 2017 at 3:19 PM, work ran <work.greeran@gmail.com> wrote:
hi i am running syslog-ng with sqlite i am sending 50 logs per second and i am getting 80% cpu usage. could someone tell me how i can fix this or is it normal thanks Ran
syslog-ng 3.2.5 Installer-Version: 3.2.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng -ose--mainline--3.2#master#9d4bea28198bd731df1a61e980a2af5b88d81116 Compile-Date: Jul 25 2014 15:20:50 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: on Enable-SSL: off Enable-SQL: on Enable-Linux-Caps: off Enable-Pcre: on Enable-Pacct: off
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support /documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product= syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
____________________________________________________________ __________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/? product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Balazs Scheidler
-
Fekete, Róbert
-
Scheidler, Balázs
-
work ran