<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="generator" content="Osso Notes">
<title></title></head>
<body>
<p>hi,
<br>
<br>----- Original message -----
<br>> My current issue:
<br>>
<br>> syslog ~ % watch -d 'sudo syslog-ng-ctl stats | sort -rnk2 -t ";" | grep
<br>> "_custom"'
<br>>
<br>> dst.sql;d_mysql_example_custom#0;mysql,10.0.0.1,3306,syslog_production,custom_example_${HO;a;stored;
<br>> *1000*
<br>> dst.sql;d_mysql_example_custom#0;mysql,10.0.0.1,3306,syslog_production,custom_example_${HO;a;dropped;0
<br>> dst.file;d_app_example_custom#0;/logs/example/custom.log;o;stored;0
<br>> dst.file;d_app_example_custom#0;/logs/example/custom.log;o;processed;351305
<br>> dst.file;d_app_example_custom#0;/logs/example/custom.log;o;dropped;0
<br>> destination;d_mysql_example_custom;;a;processed;331953
<br>> destination;d_app_example_custom;;a;processed;351305
<br>>
<br>> It just stops to read the source after a random time(1-2-3hours) with
<br>> 1000 stored statements. There are no problems at mysql destination. My
<br>> current configuration: <a href="https://gist.github.com/9f5619573d2f3e9f071c">https://gist.github.com/9f5619573d2f3e9f071c</a>
<br>
<br>for some reasons sql destination has stalled, as it seems. though I've not seen such issues recently.
<br>
<br>>
<br>> I've already tried to tune all the values, it doesn't seem to help.
<br>>
<br>
<br>this seems to be a bug in the sql destination. 1000 seems to be the window size for your source, the queue becomes filled, but then the sql destination doesn't flush messages.
<br>
<br>or does it?
<br>
<br>it might also happen that it's slow. syslog-ng maxes out the queue, then stops until messages are emptied. once there are free slots it starts again: fills it up, stalls.
<br>
<br>> Also I'm not able to enable debug logs due to
<br>> <a href="https://bugzilla.balabit.com/show_bug.cgi?id=208">https://bugzilla.balabit.com/show_bug.cgi?id=208</a>
<br>>
<br>>
<br>>
<br>>
<br>> On Mon, Nov 5, 2012 at 2:32 PM, Gergely Nagy <<a href="mailto:algernon@balabit.hu">algernon@balabit.hu</a>> wrote:
<br>>
<br>> > Anton Koldaev <<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a>> writes:
<br>> >
<br>> > > I wanted to know if syslog-ng developers has some tools like
<br>> > > mysqltuner
<br>> > or
<br>> > > just a shell scripts to check syslog-ng configuration and get some
<br>> > > recommendations on tuning?
<br>> >
<br>> > My bottleneck is usually not syslog-ng, so I use perf/tuning tools to
<br>> > whatever is on the other end (be that a database, filesystem or
<br>> > network). To see how much I need to tune the various syslog-ng buffers,
<br>> > I do load testing in a simulated environment, and base my settings on
<br>> > the number of dropped messages, and tune both the receiving end and
<br>> > syslog-ng until the drop count gets to zero during peak-like loads.
<br>> >
<br>> > So far, this method worked remarkably well, but most of my setups have
<br>> > reasonably low incoming log volume, most time is spent post-processing
<br>> > them, which I usually do outside of syslog-ng.
<br>> >
<br>> > > For example if I'm using flow-control+multiple destinations it can
<br>> > > stop reading the source at any time and I have no idea when and why
<br>> > > it's happening and which value should I tune.
<br>> >
<br>> > It would be nice if syslog-ng would log an info (so that I don't need
<br>> > to enable debug logging on a live system) level message when
<br>> > flow-control kicks in (and when it stops). For bonus points, if it
<br>> > could tell what triggered it, and which source it applies to, that'd
<br>> > be great.
<br>> >
<br>> > I don't think we can do this yet, though.
<br>> >
<br>> > --
<br>> > |8]
<br>> >
<br>> >
<br>> > ______________________________________________________________________________
<br>> > Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
<br>> > Documentation:
<br>> > <a href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
<br>> > FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
<br>> >
<br>> >
<br>>
<br>>
<br>> --
<br>> Best regards,
<br>> Koldaev Anton
<br><br></p>
</body>
</html>