[syslog-ng] suitability of syslog-ng with long messages

[Balazs Scheidler]

On Mon, 2008-06-23 at 12:59 -0400, Joe Shaw wrote:
> Hi,
> 
> I am working on a project in which we will have many gigabytes of data
> logged per day.  The log producers themselves were mostly developed in
> house and presently are simply logging to a file, but as we build out
> the system we'll obviously need some centralization.  I've been
> thinking about using syslog-ng for this.
> 
> While most of the messages are relatively small -- on the order of
> hundreds of bytes or a couple kilobytes -- there are some exceptional
> cases where we are generating messages that could be as large as two
> megabytes and need these to be kept in tact.
> 
> We'll be using the tcp() destination driver.  Are there any
> performance considerations, gotchas, etc. in setting
> log_msg_size(2097152)?  Would that have a bad interaction memory
> usage-wise, for example, with log_fifo_size() if it were set to 1000?
> I am trying to get a feel if syslog-ng is the right tool for the
> problem here, or if something else would be more suitable.
> 
> Thanks for any info you can provide,

The input buffer for every incoming connection is sized according to
log_msg_size(), thus if you have 2MB maximum size, then your input
buffer usage will be 2MB/connection.

Additionally, the tcp() driver terminates records by using the NL
character, searching the NL character in 2MB of data might be a bit
slow. I'm working on the next version of syslog-ng that adds
syslog-protocol-transport-tls style framing, which prepends the message
size to each message, that'd scale better for large sizes like this.

The other question is FIFO size, like you mentioned. The worst case
scenario is that need have 2MB * log_fifo_size() memory for each
destination. It is currently not possible to limit fifo size by bytes,
and setting the fifo size to a low value can cause problems when network
is down. The disk based buffering in the Premium Edition could help
though (in that case a small number of messages are stored in memory the
rest goes to disk).

All in all, I'd wait for the framed transport with the next syslog-ng
version and maybe would use the Premium Edition if budgets allow that. 

-- 
Bazsi