[syslog-ng] Flow-control

[Evan Rempel]

There is another huge difference

You have to consider why any control is needed. It is because the *destination* can not accept the message fast enough.

Without flow control...

Your syslog server is attempting to handle messages faster than the *destination* will accept.
Syslog buffers fill.
syslog starts to drop messages.

Sounds straight forward, however, when the central log server drops messages it drops messages FROM ANY AND ALL SERVERS that send syslog messages to it. One host flooding messages will result in dropped messages for all hosts.


Now with flow control

Your syslog server is attempting to handle messages faster than the *destination* will accept.
Syslog buffers fill.
Syslog stop reading the source (OS buffers)
OS buffers fill (no dropping with TCP)
remote hosts start buffering messages.
central server starts reading its source (log_fetch_limit from each connection)

Because the connection are processed in a round robin fashion, there is a kind of fair share being used.
As long as the number of actively logging servers (A) does not exceed A/(A+1) ratio of the max destination rate,
then only the server flooding the log stream will drop messages.

In practice a single host that is flooding log lines will drop messages while all other hosts will continue to operate normally.







________________________________________
From: syslog-ng-bounces at lists.balabit.hu [syslog-ng-bounces at lists.balabit.hu] on behalf of Anton Koldaev [koldaevav at gmail.com]
Sent: Tuesday, January 15, 2013 11:16 PM
To: Syslog-ng users' and developers' mailing list; Balazs Scheidler
Subject: Re: [syslog-ng] Flow-control

I talked to algernon in IRC yesterday:
[14:33] <iroller_> algernon: could you explain please what happens when syslog-ng stops reading the source(flow-control enabled)? As I understand if the source is "file()" it will stop reading it at current position and will continue later at the same postition, right? What if I have tcp source where my app is sending logs all the time?
[14:39] <algernon> iroller_: in case of tcp(), it will stop reading from there, kernel buffers will fill up, and the remote end will slow down. if messages pile up there, and fill the sending syslogd's buffers, then they'll likely be dropped on that side
[14:40] <iroller_> algernon: so by enabling flow-control we're just switching from syslog-ng's fifo buffer to kernel buffers, right?
[14:41] <algernon> as far as I understand, yes. (I'm not using flow control anywhere, and my knowledge of how it works in syslog-ng is a bit rusty, unfortunately)
[14:44] <iroller_> we need more balabit guys here :)


Bazsi, could you please give some more information on it? What's the purpose of switching from fifo to kernel buffers?


On Tue, Jan 8, 2013 at 4:38 PM, Anton Koldaev <koldaevav at gmail.com<mailto:koldaevav at gmail.com>> wrote:
For example at one moment of time I see the following values(with flow-control disabled):
dst_syslog.total.stored:      10000 (msg)
dst_syslog.total.dropped:    12179 (msg per min)
dst_syslog.total.processed: 183800 (msg per min)

How should flow-control help me here?


On Tue, Jan 8, 2013 at 4:15 PM, Anton Koldaev <koldaevav at gmail.com<mailto:koldaevav at gmail.com>> wrote:
As I understand syslog-ng will buffer the lines in buffer until it can process them, right? Which buffer?


On Tue, Jan 8, 2013 at 4:10 PM, Anton Koldaev <koldaevav at gmail.com<mailto:koldaevav at gmail.com>> wrote:
Could you please explain the following statement:
> If the control window is full, syslog-ng stops reading messages from the source until some messages are successfully sent to the destination.

What does that mean - "stops reading messages from the source"? My applications is still sending messages to this souce so where will all the logs at that moment?
Where will it start reading the source? From the same point it stopped or not?

--
Best regards,
Koldaev Anton



--
Best regards,
Koldaev Anton



--
Best regards,
Koldaev Anton



--
Best regards,
Koldaev Anton