----- Original message -----
I thing it would be very useful to use the statistics to determine how close the buffering has gotten to the limit of the log_fifo_size before messages start being dropped. This information can be used to;
hmm. IIRC there's a stored counter that measures how much messages are stored in a queue. I'm unable to check the source right now, and this is only a vague recollection, or might be a pe feature I stumbled upon.
- proactively change the configuration on a host. - calculate the rate of buffer consumption and predict how much time will elapse before dropping will start. Alerts to system operators can be created prior to messages being dropped.
This information can be calculated from having the number of messages queued, and the log_fifo_size for the destination, or a percentage full could be used.
I would prefer the log_fifo_size.
I would also prefer to see all of the stats for a destination on a single line. Programaticaly it is easier to split the numbers up than it is the try to join them back together based on the first three fields of a stat line
dst.file;d_var_syslog#0;/var/log/syslog.20120908.000000;a;dropped;0 dst.file;d_var_syslog#0;/var/log/syslog.20120908.000000;a;processed;326 dst.file;d_var_syslog#0;/var/log/syslog.20120908.000000;a;stored;0
huh, this is the stored counter I was referring to above. so it definitely exists, the question remains if it gets updated properly.
Something like
dst.file;d_var_syslog#0;/var/log/syslog.20120908.000000;a;0;326;0;5000,1347143526
where the 4 numbers are dropped, processed, stored, fifo_size and stamp or two different formats, one for source and one for destination
src.file;local#2;/proc/kmsg;a;1549;1347143526 dst.file;d_var_syslog#0;/var/log/syslog.20120908.000000;a;0;326;0;5000
this could probably be done, although a little problematic to extend, and would break applications that relied on the current format. perhaps a separate stats switch could trigger this alternate format.
where src has processed and stamp dst has dropped, processed, stored, fifo_size
Possibly the source could also show the flow-control state as well, or even better, the flow control duration, so if a source has been blocked for x number of seconds I can alert an operations center about the condition.
like the longest 'blocking period' since the start of syslog-ng. this could be monitored easily.
Thanks again for your consideration.
good ideas in general, and shouldn't be difficult either. I'm somewhat overwhelmed these days.