[syslog-ng] Flow-control

[Anton Koldaev]

Evan,
I have a syslog-ng installed on each client host to receive messages from
the app/db installed on the same host and central syslog-ng that receives
all the logs messages from clients and puts them into files(mostly). I see
no dropped messages at central syslog-ng server but there are a lot of
drops at my clients(not all the time, but periodically).
When my central server is unable to handle the load(mostly because of disk
IO) I see drops in client's syslog-ng stats.
Is there any reason to use flow-control at clients or central server in
such scheme?


On Wed, Jan 16, 2013 at 11:58 AM, Evan Rempel <erempel at uvic.ca> wrote:

> There is another huge difference
>
> You have to consider why any control is needed. It is because the
> *destination* can not accept the message fast enough.
>
> Without flow control...
>
> Your syslog server is attempting to handle messages faster than the
> *destination* will accept.
> Syslog buffers fill.
> syslog starts to drop messages.
>
> Sounds straight forward, however, when the central log server drops
> messages it drops messages FROM ANY AND ALL SERVERS that send syslog
> messages to it. One host flooding messages will result in dropped messages
> for all hosts.
>
>
> Now with flow control
>
> Your syslog server is attempting to handle messages faster than the
> *destination* will accept.
> Syslog buffers fill.
> Syslog stop reading the source (OS buffers)
> OS buffers fill (no dropping with TCP)
> remote hosts start buffering messages.
> central server starts reading its source (log_fetch_limit from each
> connection)
>
> Because the connection are processed in a round robin fashion, there is a
> kind of fair share being used.
> As long as the number of actively logging servers (A) does not exceed
> A/(A+1) ratio of the max destination rate,
> then only the server flooding the log stream will drop messages.
>
> In practice a single host that is flooding log lines will drop messages
> while all other hosts will continue to operate normally.
>
>
>
>
>
>
>
> ________________________________________
> From: syslog-ng-bounces at lists.balabit.hu [
> syslog-ng-bounces at lists.balabit.hu] on behalf of Anton Koldaev [
> koldaevav at gmail.com]
> Sent: Tuesday, January 15, 2013 11:16 PM
> To: Syslog-ng users' and developers' mailing list; Balazs Scheidler
> Subject: Re: [syslog-ng] Flow-control
>
> I talked to algernon in IRC yesterday:
> [14:33] <iroller_> algernon: could you explain please what happens when
> syslog-ng stops reading the source(flow-control enabled)? As I understand
> if the source is "file()" it will stop reading it at current position and
> will continue later at the same postition, right? What if I have tcp source
> where my app is sending logs all the time?
> [14:39] <algernon> iroller_: in case of tcp(), it will stop reading from
> there, kernel buffers will fill up, and the remote end will slow down. if
> messages pile up there, and fill the sending syslogd's buffers, then
> they'll likely be dropped on that side
> [14:40] <iroller_> algernon: so by enabling flow-control we're just
> switching from syslog-ng's fifo buffer to kernel buffers, right?
> [14:41] <algernon> as far as I understand, yes. (I'm not using flow
> control anywhere, and my knowledge of how it works in syslog-ng is a bit
> rusty, unfortunately)
> [14:44] <iroller_> we need more balabit guys here :)
>
>
> Bazsi, could you please give some more information on it? What's the
> purpose of switching from fifo to kernel buffers?
>
>
> On Tue, Jan 8, 2013 at 4:38 PM, Anton Koldaev <koldaevav at gmail.com<mailto:
> koldaevav at gmail.com>> wrote:
> For example at one moment of time I see the following values(with
> flow-control disabled):
> dst_syslog.total.stored:      10000 (msg)
> dst_syslog.total.dropped:    12179 (msg per min)
> dst_syslog.total.processed: 183800 (msg per min)
>
> How should flow-control help me here?
>
>
> On Tue, Jan 8, 2013 at 4:15 PM, Anton Koldaev <koldaevav at gmail.com<mailto:
> koldaevav at gmail.com>> wrote:
> As I understand syslog-ng will buffer the lines in buffer until it can
> process them, right? Which buffer?
>
>
> On Tue, Jan 8, 2013 at 4:10 PM, Anton Koldaev <koldaevav at gmail.com<mailto:
> koldaevav at gmail.com>> wrote:
> Could you please explain the following statement:
> > If the control window is full, syslog-ng stops reading messages from the
> source until some messages are successfully sent to the destination.
>
> What does that mean - "stops reading messages from the source"? My
> applications is still sending messages to this souce so where will all the
> logs at that moment?
> Where will it start reading the source? From the same point it stopped or
> not?
>
> --
> Best regards,
> Koldaev Anton
>
>
>
> --
> Best regards,
> Koldaev Anton
>
>
>
> --
> Best regards,
> Koldaev Anton
>
>
>
> --
> Best regards,
> Koldaev Anton
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>


-- 
Best regards,
Koldaev Anton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130116/4e52584e/attachment-0001.htm