[syslog-ng] Flow-control
Evan Rempel
erempel at uvic.ca
Wed Jan 16 17:41:26 CET 2013
The easy answer is "make usre you have a dik system that can handle the load".
A more useful answer is;
1. Use relatively fast disk. We use a caching hardware raid 5 controller that front ends 4 SAS 15k drives. Nothing to expensive
2. Tune syslog-ng to be as easy on disk as possible. flush_lines(15000); flush_timeout(2000);
3. Fewer files is better than lots of files. For instance I do not advocate splitting the syslog stream up by host. In my environment I would have
approx 5,000 files. That means that when flush_timeout expires, 5,000 IOs to the disk are needed. If it all went into 1 file, it could be
just 1 very large IO which would be sequential.
4. Tune the syslog server to buffer a significant amount
log_fifo_size(20000000); log_iw_size(1000000); log_fetch_limit(20000);
5. enable flow-control on the syslog-server
6. Tune the syslog client to buffer a reasonable amount;
unix-dgram("/dev/log" max_connections(10000) log_iw_size(1000000) );
log_fifo_size(4000000);
7. Tune syslog client to be somewhat network efficient.
flush_lines(500);
flush_timeout(1000);
8. Disable flow control on the syslog client.
This my be controversial, but if syslog-ng stops reading the /dev/log device, applications that log via syslog can end up blocking
on the syslog call. In those cases services are affected. Personally I would rather drop a few log lines than take out a client facing
service.
With this we easily handle 1,000,000 message per minute (peak per second rates of 60,000) without dropping any messages.
Hope this is useful.
Evan.
On 01/16/2013 01:51 AM, Anton Koldaev wrote:
> Evan,
> I have a syslog-ng installed on each client host to receive messages from the app/db installed on the same host and central syslog-ng that receives all the logs messages from clients and puts them into files(mostly). I see no dropped messages at central syslog-ng server but there are a lot of drops at my clients(not all the time, but periodically).
> When my central server is unable to handle the load(mostly because of disk IO) I see drops in client's syslog-ng stats.
> Is there any reason to use flow-control at clients or central server in such scheme?
>
>
> On Wed, Jan 16, 2013 at 11:58 AM, Evan Rempel <erempel at uvic.ca <mailto:erempel at uvic.ca>> wrote:
>
> There is another huge difference
>
> You have to consider why any control is needed. It is because the *destination* can not accept the message fast enough.
>
> Without flow control...
>
> Your syslog server is attempting to handle messages faster than the *destination* will accept.
> Syslog buffers fill.
> syslog starts to drop messages.
>
> Sounds straight forward, however, when the central log server drops messages it drops messages FROM ANY AND ALL SERVERS that send syslog messages to it. One host flooding messages will result in dropped messages for all hosts.
>
>
> Now with flow control
>
> Your syslog server is attempting to handle messages faster than the *destination* will accept.
> Syslog buffers fill.
> Syslog stop reading the source (OS buffers)
> OS buffers fill (no dropping with TCP)
> remote hosts start buffering messages.
> central server starts reading its source (log_fetch_limit from each connection)
>
> Because the connection are processed in a round robin fashion, there is a kind of fair share being used.
> As long as the number of actively logging servers (A) does not exceed A/(A+1) ratio of the max destination rate,
> then only the server flooding the log stream will drop messages.
>
> In practice a single host that is flooding log lines will drop messages while all other hosts will continue to operate normally.
>
>
>
>
>
>
>
> ________________________________________
> From: syslog-ng-bounces at lists.balabit.hu <mailto:syslog-ng-bounces at lists.balabit.hu> [syslog-ng-bounces at lists.balabit.hu <mailto:syslog-ng-bounces at lists.balabit.hu>] on behalf of Anton Koldaev [koldaevav at gmail.com <mailto:koldaevav at gmail.com>]
> Sent: Tuesday, January 15, 2013 11:16 PM
> To: Syslog-ng users' and developers' mailing list; Balazs Scheidler
> Subject: Re: [syslog-ng] Flow-control
>
> I talked to algernon in IRC yesterday:
> [14:33] <iroller_> algernon: could you explain please what happens when syslog-ng stops reading the source(flow-control enabled)? As I understand if the source is "file()" it will stop reading it at current position and will continue later at the same postition, right? What if I have tcp source where my app is sending logs all the time?
> [14:39] <algernon> iroller_: in case of tcp(), it will stop reading from there, kernel buffers will fill up, and the remote end will slow down. if messages pile up there, and fill the sending syslogd's buffers, then they'll likely be dropped on that side
> [14:40] <iroller_> algernon: so by enabling flow-control we're just switching from syslog-ng's fifo buffer to kernel buffers, right?
> [14:41] <algernon> as far as I understand, yes. (I'm not using flow control anywhere, and my knowledge of how it works in syslog-ng is a bit rusty, unfortunately)
> [14:44] <iroller_> we need more balabit guys here :)
>
>
> Bazsi, could you please give some more information on it? What's the purpose of switching from fifo to kernel buffers?
>
>
> On Tue, Jan 8, 2013 at 4:38 PM, Anton Koldaev <koldaevav at gmail.com <mailto:koldaevav at gmail.com><mailto:koldaevav at gmail.com <mailto:koldaevav at gmail.com>>> wrote:
> For example at one moment of time I see the following values(with flow-control disabled):
> dst_syslog.total.stored: 10000 (msg)
> dst_syslog.total.dropped: 12179 (msg per min)
> dst_syslog.total.processed: 183800 (msg per min)
>
> How should flow-control help me here?
>
>
> On Tue, Jan 8, 2013 at 4:15 PM, Anton Koldaev <koldaevav at gmail.com <mailto:koldaevav at gmail.com><mailto:koldaevav at gmail.com <mailto:koldaevav at gmail.com>>> wrote:
> As I understand syslog-ng will buffer the lines in buffer until it can process them, right? Which buffer?
>
>
> On Tue, Jan 8, 2013 at 4:10 PM, Anton Koldaev <koldaevav at gmail.com <mailto:koldaevav at gmail.com><mailto:koldaevav at gmail.com <mailto:koldaevav at gmail.com>>> wrote:
> Could you please explain the following statement:
> > If the control window is full, syslog-ng stops reading messages from the source until some messages are successfully sent to the destination.
>
> What does that mean - "stops reading messages from the source"? My applications is still sending messages to this souce so where will all the logs at that moment?
> Where will it start reading the source? From the same point it stopped or not?
>
> --
> Best regards,
> Koldaev Anton
>
>
>
> --
> Best regards,
> Koldaev Anton
>
>
>
> --
> Best regards,
> Koldaev Anton
>
>
>
> --
> Best regards,
> Koldaev Anton
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
>
> --
> Best regards,
> Koldaev Anton
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
--
Evan Rempel erempel at uvic.ca
Senior Systems Administrator 250.721.7691
Data Centre Services, University Systems, University of Victoria
More information about the syslog-ng
mailing list