<div dir="ltr">Evan,<div>I have a syslog-ng installed on each client host to receive messages from the app/db installed on the same host and central syslog-ng that receives all the logs messages from clients and puts them into files(mostly). I see no dropped messages at central syslog-ng server but there are a lot of drops at my clients(not all the time, but periodically).</div>
<div style>When my central server is unable to handle the load(mostly because of disk IO) I see drops in client's syslog-ng stats.</div><div style>Is there any reason to use flow-control at clients or central server in such scheme?</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Jan 16, 2013 at 11:58 AM, Evan Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
There is another huge difference<br>
<br>
You have to consider why any control is needed. It is because the *destination* can not accept the message fast enough.<br>
<br>
Without flow control...<br>
<br>
Your syslog server is attempting to handle messages faster than the *destination* will accept.<br>
Syslog buffers fill.<br>
syslog starts to drop messages.<br>
<br>
Sounds straight forward, however, when the central log server drops messages it drops messages FROM ANY AND ALL SERVERS that send syslog messages to it. One host flooding messages will result in dropped messages for all hosts.<br>
<br>
<br>
Now with flow control<br>
<br>
Your syslog server is attempting to handle messages faster than the *destination* will accept.<br>
Syslog buffers fill.<br>
Syslog stop reading the source (OS buffers)<br>
OS buffers fill (no dropping with TCP)<br>
remote hosts start buffering messages.<br>
central server starts reading its source (log_fetch_limit from each connection)<br>
<br>
Because the connection are processed in a round robin fashion, there is a kind of fair share being used.<br>
As long as the number of actively logging servers (A) does not exceed A/(A+1) ratio of the max destination rate,<br>
then only the server flooding the log stream will drop messages.<br>
<br>
In practice a single host that is flooding log lines will drop messages while all other hosts will continue to operate normally.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
________________________________________<br>
From: <a href="mailto:syslog-ng-bounces@lists.balabit.hu">syslog-ng-bounces@lists.balabit.hu</a> [<a href="mailto:syslog-ng-bounces@lists.balabit.hu">syslog-ng-bounces@lists.balabit.hu</a>] on behalf of Anton Koldaev [<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a>]<br>
Sent: Tuesday, January 15, 2013 11:16 PM<br>
To: Syslog-ng users' and developers' mailing list; Balazs Scheidler<br>
Subject: Re: [syslog-ng] Flow-control<br>
<div class="im"><br>
I talked to algernon in IRC yesterday:<br>
[14:33] <iroller_> algernon: could you explain please what happens when syslog-ng stops reading the source(flow-control enabled)? As I understand if the source is "file()" it will stop reading it at current position and will continue later at the same postition, right? What if I have tcp source where my app is sending logs all the time?<br>
[14:39] <algernon> iroller_: in case of tcp(), it will stop reading from there, kernel buffers will fill up, and the remote end will slow down. if messages pile up there, and fill the sending syslogd's buffers, then they'll likely be dropped on that side<br>
[14:40] <iroller_> algernon: so by enabling flow-control we're just switching from syslog-ng's fifo buffer to kernel buffers, right?<br>
[14:41] <algernon> as far as I understand, yes. (I'm not using flow control anywhere, and my knowledge of how it works in syslog-ng is a bit rusty, unfortunately)<br>
[14:44] <iroller_> we need more balabit guys here :)<br>
<br>
<br>
Bazsi, could you please give some more information on it? What's the purpose of switching from fifo to kernel buffers?<br>
<br>
<br>
</div><div class="im">On Tue, Jan 8, 2013 at 4:38 PM, Anton Koldaev <<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a><mailto:<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a>>> wrote:<br>
For example at one moment of time I see the following values(with flow-control disabled):<br>
dst_syslog.total.stored: 10000 (msg)<br>
dst_syslog.total.dropped: 12179 (msg per min)<br>
dst_syslog.total.processed: 183800 (msg per min)<br>
<br>
How should flow-control help me here?<br>
<br>
<br>
</div><div class="im">On Tue, Jan 8, 2013 at 4:15 PM, Anton Koldaev <<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a><mailto:<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a>>> wrote:<br>
As I understand syslog-ng will buffer the lines in buffer until it can process them, right? Which buffer?<br>
<br>
<br>
</div><div class="im">On Tue, Jan 8, 2013 at 4:10 PM, Anton Koldaev <<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a><mailto:<a href="mailto:koldaevav@gmail.com">koldaevav@gmail.com</a>>> wrote:<br>
Could you please explain the following statement:<br>
> If the control window is full, syslog-ng stops reading messages from the source until some messages are successfully sent to the destination.<br>
<br>
What does that mean - "stops reading messages from the source"? My applications is still sending messages to this souce so where will all the logs at that moment?<br>
Where will it start reading the source? From the same point it stopped or not?<br>
<br>
--<br>
Best regards,<br>
Koldaev Anton<br>
<br>
<br>
<br>
--<br>
Best regards,<br>
Koldaev Anton<br>
<br>
<br>
<br>
--<br>
Best regards,<br>
Koldaev Anton<br>
<br>
<br>
<br>
--<br>
Best regards,<br>
Koldaev Anton<br>
</div>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br>Best regards,<br>Koldaev Anton
</div>