hi, ----- Original message -----
My current issue:
syslog ~ % watch -d 'sudo syslog-ng-ctl stats | sort -rnk2 -t ";" | grep "_custom"'
dst.sql;d_mysql_example_custom#0;mysql,10.0.0.1,3306,syslog_production,custom_example_${HO;a;stored; *1000* dst.sql;d_mysql_example_custom#0;mysql,10.0.0.1,3306,syslog_production,custom_example_${HO;a;dropped;0 dst.file;d_app_example_custom#0;/logs/example/custom.log;o;stored;0 dst.file;d_app_example_custom#0;/logs/example/custom.log;o;processed;351305 dst.file;d_app_example_custom#0;/logs/example/custom.log;o;dropped;0 destination;d_mysql_example_custom;;a;processed;331953 destination;d_app_example_custom;;a;processed;351305
It just stops to read the source after a random time(1-2-3hours) with 1000 stored statements. There are no problems at mysql destination. My current configuration: https://gist.github.com/9f5619573d2f3e9f071c
for some reasons sql destination has stalled, as it seems. though I've not seen such issues recently.
I've already tried to tune all the values, it doesn't seem to help.
this seems to be a bug in the sql destination. 1000 seems to be the window size for your source, the queue becomes filled, but then the sql destination doesn't flush messages. or does it? it might also happen that it's slow. syslog-ng maxes out the queue, then stops until messages are emptied. once there are free slots it starts again: fills it up, stalls.
Also I'm not able to enable debug logs due to https://bugzilla.balabit.com/show_bug.cgi?id=208
On Mon, Nov 5, 2012 at 2:32 PM, Gergely Nagy <algernon@balabit.hu> wrote:
Anton Koldaev <koldaevav@gmail.com> writes:
I wanted to know if syslog-ng developers has some tools like mysqltuner or just a shell scripts to check syslog-ng configuration and get some recommendations on tuning?
My bottleneck is usually not syslog-ng, so I use perf/tuning tools to whatever is on the other end (be that a database, filesystem or network). To see how much I need to tune the various syslog-ng buffers, I do load testing in a simulated environment, and base my settings on the number of dropped messages, and tune both the receiving end and syslog-ng until the drop count gets to zero during peak-like loads.
So far, this method worked remarkably well, but most of my setups have reasonably low incoming log volume, most time is spent post-processing them, which I usually do outside of syslog-ng.
For example if I'm using flow-control+multiple destinations it can stop reading the source at any time and I have no idea when and why it's happening and which value should I tune.
It would be nice if syslog-ng would log an info (so that I don't need to enable debug logging on a live system) level message when flow-control kicks in (and when it stops). For bonus points, if it could tell what triggered it, and which source it applies to, that'd be great.
I don't think we can do this yet, though.
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Best regards, Koldaev Anton