[syslog-ng] Debugging tools for syslog-ng

[Balazs Scheidler]

hi,

----- Original message -----
> My current issue:
> 
> syslog ~ % watch -d 'sudo syslog-ng-ctl stats | sort -rnk2 -t ";" | grep
> "_custom"'
> 
> dst.sql;d_mysql_example_custom#0;mysql,10.0.0.1,3306,syslog_production,custom_example_${HO;a;stored;
> *1000*
> dst.sql;d_mysql_example_custom#0;mysql,10.0.0.1,3306,syslog_production,custom_example_${HO;a;dropped;0
> dst.file;d_app_example_custom#0;/logs/example/custom.log;o;stored;0
> dst.file;d_app_example_custom#0;/logs/example/custom.log;o;processed;351305
> dst.file;d_app_example_custom#0;/logs/example/custom.log;o;dropped;0
> destination;d_mysql_example_custom;;a;processed;331953
> destination;d_app_example_custom;;a;processed;351305
> 
> It just stops to read the source after a random time(1-2-3hours) with
> 1000 stored statements. There are no problems at mysql destination. My
> current configuration: https://gist.github.com/9f5619573d2f3e9f071c

for some reasons sql destination has stalled, as it seems. though I've not seen such issues recently.

> 
> I've already tried to tune all the values, it doesn't seem to help.
> 

this seems to be a bug in the sql destination. 1000 seems to be the window size for your source, the queue becomes filled, but then the sql destination doesn't flush messages.

or does it?

it might also happen that it's slow. syslog-ng maxes out the queue, then stops until messages are emptied. once there are free slots it starts again: fills it up, stalls.

> Also I'm not able to enable debug logs due to
> https://bugzilla.balabit.com/show_bug.cgi?id=208
> 
> 
> 
> 
> On Mon, Nov 5, 2012 at 2:32 PM, Gergely Nagy <algernon at balabit.hu> wrote:
> 
> > Anton Koldaev <koldaevav at gmail.com> writes:
> > 
> > > I wanted to know if syslog-ng developers has some tools like
> > > mysqltuner
> > or
> > > just a shell scripts to check syslog-ng configuration and get some
> > > recommendations on tuning?
> > 
> > My bottleneck is usually not syslog-ng, so I use perf/tuning tools to
> > whatever is on the other end (be that a database, filesystem or
> > network). To see how much I need to tune the various syslog-ng buffers,
> > I do load testing in a simulated environment, and base my settings on
> > the number of dropped messages, and tune both the receiving end and
> > syslog-ng until the drop count gets to zero during peak-like loads.
> > 
> > So far, this method worked remarkably well, but most of my setups have
> > reasonably low incoming log volume, most time is spent post-processing
> > them, which I usually do outside of syslog-ng.
> > 
> > > For example if I'm using flow-control+multiple destinations it can
> > > stop reading the source at any time and I have no idea when and why
> > > it's happening and which value should I tune.
> > 
> > It would be nice if syslog-ng would log an info (so that I don't need
> > to enable debug logging on a live system) level message when
> > flow-control kicks in (and when it stops). For bonus points, if it
> > could tell what triggered it, and which source it applies to, that'd
> > be great.
> > 
> > I don't think we can do this yet, though.
> > 
> > --
> > |8]
> > 
> > 
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> > http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> > 
> > 
> 
> 
> -- 
> Best regards,
> Koldaev Anton

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20121107/fb06aeda/attachment.htm