On Mon, 2008-06-23 at 12:59 -0400, Joe Shaw wrote:
Hi,
I am working on a project in which we will have many gigabytes of data logged per day. The log producers themselves were mostly developed in house and presently are simply logging to a file, but as we build out the system we'll obviously need some centralization. I've been thinking about using syslog-ng for this.
While most of the messages are relatively small -- on the order of hundreds of bytes or a couple kilobytes -- there are some exceptional cases where we are generating messages that could be as large as two megabytes and need these to be kept in tact.
We'll be using the tcp() destination driver. Are there any performance considerations, gotchas, etc. in setting log_msg_size(2097152)? Would that have a bad interaction memory usage-wise, for example, with log_fifo_size() if it were set to 1000? I am trying to get a feel if syslog-ng is the right tool for the problem here, or if something else would be more suitable.
Thanks for any info you can provide,
The input buffer for every incoming connection is sized according to log_msg_size(), thus if you have 2MB maximum size, then your input buffer usage will be 2MB/connection. Additionally, the tcp() driver terminates records by using the NL character, searching the NL character in 2MB of data might be a bit slow. I'm working on the next version of syslog-ng that adds syslog-protocol-transport-tls style framing, which prepends the message size to each message, that'd scale better for large sizes like this. The other question is FIFO size, like you mentioned. The worst case scenario is that need have 2MB * log_fifo_size() memory for each destination. It is currently not possible to limit fifo size by bytes, and setting the fifo size to a low value can cause problems when network is down. The disk based buffering in the Premium Edition could help though (in that case a small number of messages are stored in memory the rest goes to disk). All in all, I'd wait for the framed transport with the next syslog-ng version and maybe would use the Premium Edition if budgets allow that. -- Bazsi