suitability of syslog-ng with long messages
Hi, I am working on a project in which we will have many gigabytes of data logged per day. The log producers themselves were mostly developed in house and presently are simply logging to a file, but as we build out the system we'll obviously need some centralization. I've been thinking about using syslog-ng for this. While most of the messages are relatively small -- on the order of hundreds of bytes or a couple kilobytes -- there are some exceptional cases where we are generating messages that could be as large as two megabytes and need these to be kept in tact. We'll be using the tcp() destination driver. Are there any performance considerations, gotchas, etc. in setting log_msg_size(2097152)? Would that have a bad interaction memory usage-wise, for example, with log_fifo_size() if it were set to 1000? I am trying to get a feel if syslog-ng is the right tool for the problem here, or if something else would be more suitable. Thanks for any info you can provide, Joe
On Mon, 2008-06-23 at 12:59 -0400, Joe Shaw wrote:
Hi,
I am working on a project in which we will have many gigabytes of data logged per day. The log producers themselves were mostly developed in house and presently are simply logging to a file, but as we build out the system we'll obviously need some centralization. I've been thinking about using syslog-ng for this.
While most of the messages are relatively small -- on the order of hundreds of bytes or a couple kilobytes -- there are some exceptional cases where we are generating messages that could be as large as two megabytes and need these to be kept in tact.
We'll be using the tcp() destination driver. Are there any performance considerations, gotchas, etc. in setting log_msg_size(2097152)? Would that have a bad interaction memory usage-wise, for example, with log_fifo_size() if it were set to 1000? I am trying to get a feel if syslog-ng is the right tool for the problem here, or if something else would be more suitable.
Thanks for any info you can provide,
The input buffer for every incoming connection is sized according to log_msg_size(), thus if you have 2MB maximum size, then your input buffer usage will be 2MB/connection. Additionally, the tcp() driver terminates records by using the NL character, searching the NL character in 2MB of data might be a bit slow. I'm working on the next version of syslog-ng that adds syslog-protocol-transport-tls style framing, which prepends the message size to each message, that'd scale better for large sizes like this. The other question is FIFO size, like you mentioned. The worst case scenario is that need have 2MB * log_fifo_size() memory for each destination. It is currently not possible to limit fifo size by bytes, and setting the fifo size to a low value can cause problems when network is down. The disk based buffering in the Premium Edition could help though (in that case a small number of messages are stored in memory the rest goes to disk). All in all, I'd wait for the framed transport with the next syslog-ng version and maybe would use the Premium Edition if budgets allow that. -- Bazsi
Hi, Thanks for the info! On Tue, Jun 24, 2008 at 3:30 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
The other question is FIFO size, like you mentioned. The worst case scenario is that need have 2MB * log_fifo_size() memory for each destination. It is currently not possible to limit fifo size by bytes, and setting the fifo size to a low value can cause problems when network is down.
The worst case is 2MB * log_fifo_size(), but in an ordinary case where only some data is being buffered, and the messages are smaller than 2MB messages, are 2MB buffers still being allocated? To put it differently, is a buffer of log_msg_size() always allocated for the cached messages regardless of size, or only for the actual size of the message?
The disk based buffering in the Premium Edition could help though (in that case a small number of messages are stored in memory the rest goes to disk).
All in all, I'd wait for the framed transport with the next syslog-ng version and maybe would use the Premium Edition if budgets allow that.
How long until the next version? ;) Joe
On Tue, 2008-06-24 at 10:32 -0400, Joe Shaw wrote:
Hi,
Thanks for the info!
On Tue, Jun 24, 2008 at 3:30 AM, Balazs Scheidler <bazsi@balabit.hu> wrote:
The other question is FIFO size, like you mentioned. The worst case scenario is that need have 2MB * log_fifo_size() memory for each destination. It is currently not possible to limit fifo size by bytes, and setting the fifo size to a low value can cause problems when network is down.
The worst case is 2MB * log_fifo_size(), but in an ordinary case where only some data is being buffered, and the messages are smaller than 2MB messages, are 2MB buffers still being allocated? To put it differently, is a buffer of log_msg_size() always allocated for the cached messages regardless of size, or only for the actual size of the message?
Of course only the actual size is allocated.
The disk based buffering in the Premium Edition could help though (in that case a small number of messages are stored in memory the rest goes to disk).
All in all, I'd wait for the framed transport with the next syslog-ng version and maybe would use the Premium Edition if budgets allow that.
How long until the next version? ;)
I'm working on the final parts, but I made the same mistake as with syslog-ng OSE 2.1, instead of doing GPLd things in public I was doing them in my private branch, simply because I was too lazy to maintain two branches during development. The framing stuff is ready, it is at the QA people right now, so the next version is coming Real Soon Now. :) -- Bazsi
participants (2)
-
Balazs Scheidler
-
Joe Shaw