<span style="font-family:arial,sans-serif;font-size:13px">> this seems to be a bug in the sql destination. 1000 seems to be the window size for your source, the queue becomes filled, but then the sql destination doesn't flush messages. </span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">> or does it?</span><span style="font-family:arial,sans-serif;font-size:13px"> </span><br>Looks like it doesn't write anything after filling up to 1000. I spent a few hours on waiting for it to be flushed but got no results.<div>
log_iw_size for my source should be 20000:</div><div><pre style="margin-top:0px;margin-bottom:0px;padding:0px;font-size:12px;line-height:1.4em;font-family:'Bitstream Vera Sans Mono',Courier,monospace;color:rgb(0,0,0)">
<div class="" id="LC31" style="margin:0px;padding:0px 0px 0px 1em;line-height:1.4em"> syslog(ip(0.0.0.0) transport("tcp") port(5141) max-connections(200) log_iw_size(20000) flags("threaded") log_fetch_limit(100));</div>
</pre><br><br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px">> it might also happen that it's slow. syslog-ng maxes out the queue, then stops until messages are emptied. once there are free slots it starts again: fills it up, stalls.</span><br>
</div><div class="gmail_extra">Looks like it never gets any free slots in my configuration since I don't see any logging after the queue is maxed out for a few hours.</div><div class="gmail_extra"><br></div><div class="gmail_extra">
Could you give me any recommendations? Looks like log_iw_size just doesn't work for my source and I have no idea how to fix it.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks</div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Nov 7, 2012 at 9:38 AM, Balazs Scheidler <span dir="ltr"><<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<u></u>
<div>
<p></p><div class="im">----- Original message -----
<br>> > > > If flow-control is in use and one of the destinations cannot accept
<br>> the
<br>> > > messages, the other destinations do not receive any messages either,
<br>> > > because syslog-ng stops reading the source.
<br>
<br></div>I misunderstood what you wrote here. The docs is correct. For a single source, throttling the source side means that neither destinations receive messages.
<br><div class="im">
<br>>
<br>> > this is not true. syslog-ng stops sources individually when their
<br>> > window
<br>> is full.
<br>>
<br>> But it's a quote from
<br>> <a href="http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/index.html-single.html#id555527" target="_blank">http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-3.3-guides/syslog-ng-ose-v3.3-guide-admin-en.html/index.html-single.html#id555527</a>
<br>>
<br>>
<br>>
<br>> On Mon, Nov 5, 2012 at 9:34 AM, Balazs Scheidler <<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>>
<br>> wrote:
<br>>
<br></div>> > **
<br><div><div class="h5">> >
<br>> > ----- Original message -----
<br>> > > Thanks for your reply
<br>> > > How can I understand when it's enough to increase things? Is there
<br>> > > any manual way to get current values of each buffer, etc?
<br>> >
<br>> > well, I tend to use loggen for performance tests, also you can query
<br>> > syslog-ng internal statistics using 'syslog-ng-ctl stats'
<br>> >
<br>> > > Also since I'm logging a lot of things I'd love to know if there are
<br>> > some
<br>> > > other ways to lose messages without seeing them in "dropped"?
<br>> >
<br>> > syslog-ng counts everything it dropped using the dropped counters for
<br>> > destinations (which is a log-fifo overflow btw)
<br>> >
<br>> > messages can be lost outside syslog-ng because of transport reasons:
<br>> > * udp shouldn't be used for anything serious.
<br>> > * connection breaks can cause message loss
<br>> >
<br>> >
<br>> > >
<br>> > > > In general, performance wise you want to increase stuff
<br>> > > > (log-fetch-limit,
<br>> > > log-iw-size, flush-lines for file destinations), memory-use and
<br>> > > reliability wise you want to decrease them.
<br>> > > > Also, you have to make sure that sum(log-iw-size) < log-fifo-size.
<br>> > > So you propose just randomly tune those params? I just don't
<br>> > > understand how should I get check if it helped.
<br>> >
<br>> > no :) random tuning would be slow to converge to the ideal values.
<br>> >
<br>> >
<br>> >
<br>> > I need to see the current state of
<br>> > > each buffer(to be able to get some statistics data) to see if it
<br>> > > helps.
<br>> >
<br>> > syslog-ng-ctl stats displays the current values of statistics as a csv
<br>> > file.
<br>> >
<br>> > also you can ask syslog-ng to measure more stats by increasing
<br>> > stats-level (at the cost of some performance)
<br>> >
<br>> > >
<br>> > > And one more specific question:
<br>> > > > If flow-control is in use and one of the destinations cannot accept
<br>> > the
<br>> > > messages, the other destinations do not receive any messages either,
<br>> > > because syslog-ng stops reading the source.
<br>> >
<br>> > this is not true. syslog-ng stops sources individually when their
<br>> > window is full.
<br>> >
<br>> > > Why there is no messages about it in syslog-ng logs? It must be
<br>> > > error, don't you think so?
<br>> > > And what if I don't have flow-control enabled?
<br>> >
<br>> >
<br>>
<br>>
<br>> --
<br>> Best regards,
<br>> Koldaev Anton
<br><br></div></div><p></p>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Koldaev Anton<br>
</div>