this seems to be a bug in the sql destination. 1000 seems to be the window size for your source, the queue becomes filled, but then the sql destination doesn't flush messages. or does it? Looks like it doesn't write anything after filling up to 1000. I spent a few hours on waiting for it to be flushed but got no results. log_iw_size for my source should be 20000:
syslog(ip(0.0.0.0) transport("tcp") port(5141) max-connections(200) log_iw_size(20000) flags("threaded") log_fetch_limit(100));
it might also happen that it's slow. syslog-ng maxes out the queue, then stops until messages are emptied. once there are free slots it starts again: fills it up, stalls. Looks like it never gets any free slots in my configuration since I don't see any logging after the queue is maxed out for a few hours.
Could you give me any recommendations? Looks like log_iw_size just doesn't work for my source and I have no idea how to fix it. Thanks On Wed, Nov 7, 2012 at 9:38 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
**
----- Original message -----
If flow-control is in use and one of the destinations cannot accept the messages, the other destinations do not receive any messages either, because syslog-ng stops reading the source.
I misunderstood what you wrote here. The docs is correct. For a single source, throttling the source side means that neither destinations receive messages.
this is not true. syslog-ng stops sources individually when their window is full.
But it's a quote from
http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guide...
On Mon, Nov 5, 2012 at 9:34 AM, Balazs Scheidler <bazsi77@gmail.com> wrote:
**
----- Original message -----
Thanks for your reply How can I understand when it's enough to increase things? Is there any manual way to get current values of each buffer, etc?
well, I tend to use loggen for performance tests, also you can query syslog-ng internal statistics using 'syslog-ng-ctl stats'
Also since I'm logging a lot of things I'd love to know if there are some other ways to lose messages without seeing them in "dropped"?
syslog-ng counts everything it dropped using the dropped counters for destinations (which is a log-fifo overflow btw)
messages can be lost outside syslog-ng because of transport reasons: * udp shouldn't be used for anything serious. * connection breaks can cause message loss
In general, performance wise you want to increase stuff (log-fetch-limit, log-iw-size, flush-lines for file destinations), memory-use and reliability wise you want to decrease them. Also, you have to make sure that sum(log-iw-size) < log-fifo-size. So you propose just randomly tune those params? I just don't understand how should I get check if it helped.
no :) random tuning would be slow to converge to the ideal values.
I need to see the current state of
each buffer(to be able to get some statistics data) to see if it helps.
syslog-ng-ctl stats displays the current values of statistics as a csv file.
also you can ask syslog-ng to measure more stats by increasing stats-level (at the cost of some performance)
And one more specific question:
If flow-control is in use and one of the destinations cannot
accept
the
messages, the other destinations do not receive any messages either, because syslog-ng stops reading the source.
this is not true. syslog-ng stops sources individually when their window is full.
Why there is no messages about it in syslog-ng logs? It must be error, don't you think so? And what if I don't have flow-control enabled?
-- Best regards, Koldaev Anton
-- Best regards, Koldaev Anton