I have a remote server running syslog-ng collecting messages from network devices that are local to it and transferring them to a central server using TCP. This is to avoid messages being lost when UDP is used a a network problem occurs. All works as it should and I have tested breaking the link between the local collection station and the central server; the messages are passed to the central server when the link is restored. I had a question regarding how long the remote server will hold on to the messages when it cannot pass them to the central server. Is this a time dependent parameter, memory limit or hard coded somewhere? I couldn't find a destination driver configuration parameter for this in the documentation. Jim Mozley
On Thu, 2004-09-16 at 13:48, Jim Mozley wrote:
I have a remote server running syslog-ng collecting messages from network devices that are local to it and transferring them to a central server using TCP. This is to avoid messages being lost when UDP is used a a network problem occurs. All works as it should and I have tested breaking the link between the local collection station and the central server; the messages are passed to the central server when the link is restored.
I had a question regarding how long the remote server will hold on to the messages when it cannot pass them to the central server. Is this a time dependent parameter, memory limit or hard coded somewhere? I couldn't find a destination driver configuration parameter for this in the documentation.
syslog-ng keeps those messages in a memory buffer, whose size is controlled using the log_fifo_size() global/per-destination parameter. -- Bazsi
Balazs Scheidler wrote:
syslog-ng keeps those messages in a memory buffer, whose size is controlled using the log_fifo_size() global/per-destination parameter.
Thank you, this makes sense. As a follow-up question I'd like to ask what happens if the number of lines set by this parameter is exceeded. Are messages dropped or the oldest first overwritten? I assume this is recorded on the server that dropped the message with the STATS: dropped message? Jim Mozley
On Thu, 2004-09-16 at 17:14, Jim Mozley wrote:
Balazs Scheidler wrote:
syslog-ng keeps those messages in a memory buffer, whose size is controlled using the log_fifo_size() global/per-destination parameter.
Thank you, this makes sense.
As a follow-up question I'd like to ask what happens if the number of lines set by this parameter is exceeded.
Are messages dropped or the oldest first overwritten?
new messages are dropped
I assume this is recorded on the server that dropped the message with the STATS: dropped message?
yes -- Bazsi
Balazs Scheidler wrote:
syslog-ng keeps those messages in a memory buffer, whose size is controlled using the log_fifo_size() global/per-destination parameter.
I can do this on a global level but if I try: destination loghost { tcp("loghost" port(5140)); log_fifo_size(1000); }; The documentation showed log_fifo_size applicable to a file destination but does not list it under udp and tcp destinations. If I apply it at a global level is it only applicable to file destinations? I am using version 1.6.2 Jim Mozley
On Thu, 2004-09-16 at 18:00, Jim Mozley wrote:
Balazs Scheidler wrote:
syslog-ng keeps those messages in a memory buffer, whose size is controlled using the log_fifo_size() global/per-destination parameter.
I can do this on a global level but if I try:
destination loghost { tcp("loghost" port(5140)); log_fifo_size(1000); };
The documentation showed log_fifo_size applicable to a file destination but does not list it under udp and tcp destinations.
Hmm I can't remember whether it is applicable to tcp destinations but the syntax you used above is not correct. it should be: destination loghost { tcp("loghost" port(5140) log_fifo_size(1000)); }; -- Bazsi
Balazs Scheidler wrote:
On Thu, 2004-09-16 at 18:00, Jim Mozley wrote:
Balazs Scheidler wrote:
syslog-ng keeps those messages in a memory buffer, whose size is controlled using the log_fifo_size() global/per-destination parameter.
I can do this on a global level but if I try:
destination loghost { tcp("loghost" port(5140)); log_fifo_size(1000); };
The documentation showed log_fifo_size applicable to a file destination but does not list it under udp and tcp destinations.
Hmm I can't remember whether it is applicable to tcp destinations but the syntax you used above is not correct. it should be:
destination loghost { tcp("loghost" port(5140) log_fifo_size(1000)); };
Just tried: destination loghost { tcp("loghost" port(5140) log_fifo_size(1000)); }; Syslog-ng reports a parse error reading the configuration file, so I'm assuming this option is not supported for a tcp destination (or if it is it's taken from the global option). So I guess I'm back to my original question of finding out what the limitation is on how long/how many messages are kept for a tcp destination that is unreachable. As I understand it so far messages are kept indefinately in the buffer until its full, at which point incoming messages are dropped. So if log_fifo_size is only applicable to file destinations, then I would like to find out how to control the buffer size for tcp destinations (or at least be aware of its limitations). Jim Mozley
On Fri, 2004-09-17 at 10:31, Jim Mozley wrote:
Balazs Scheidler wrote:
On Thu, 2004-09-16 at 18:00, Jim Mozley wrote:
Balazs Scheidler wrote:
syslog-ng keeps those messages in a memory buffer, whose size is controlled using the log_fifo_size() global/per-destination parameter.
I can do this on a global level but if I try:
destination loghost { tcp("loghost" port(5140)); log_fifo_size(1000); };
The documentation showed log_fifo_size applicable to a file destination but does not list it under udp and tcp destinations.
Hmm I can't remember whether it is applicable to tcp destinations but the syntax you used above is not correct. it should be:
destination loghost { tcp("loghost" port(5140) log_fifo_size(1000)); };
Just tried:
destination loghost { tcp("loghost" port(5140) log_fifo_size(1000)); };
Syslog-ng reports a parse error reading the configuration file, so I'm assuming this option is not supported for a tcp destination (or if it is it's taken from the global option).
Checked the source, it is taken from the global option, and this patch allows it to set it for TCP/UDP targets (though the latter will never drop messages): Index: cfg-grammar.y =================================================================== RCS file: /var/cvs/syslog-ng/syslog-ng/src/cfg-grammar.y,v retrieving revision 1.58.4.4 diff -u -r1.58.4.4 cfg-grammar.y --- cfg-grammar.y 6 May 2004 08:57:52 -0000 1.58.4.4 +++ cfg-grammar.y 17 Sep 2004 09:18:38 -0000 @@ -497,7 +497,8 @@ ; dest_afinet_option - : KW_LOCALIP '(' string ')' { afinet_dest_set_localip(last_dest_driver, $3); free($3); } + : dest_item_option + | KW_LOCALIP '(' string ')' { afinet_dest_set_localip(last_dest_driver, $3); free($3); } | KW_LOCALPORT '(' NUMBER ')' { afinet_dest_set_localport(last_dest_driver, $3, NULL, NULL); } | KW_PORT '(' NUMBER ')' { afinet_dest_set_destport(last_dest_driver, $3, NULL, NULL); } | KW_DESTPORT '(' NUMBER ')' { afinet_dest_set_destport(last_dest_driver, $3, NULL, NULL); } After applying the patch, rerun make (you fill need bison installed to rebuild)
As I understand it so far messages are kept indefinately in the buffer until its full, at which point incoming messages are dropped. So if log_fifo_size is only applicable to file destinations, then I would like to find out how to control the buffer size for tcp destinations (or at least be aware of its limitations).
log_fifo_size is applicable to TCP as well, but the parser did not allow it setting on a per-destionation basis. With the patch above, it becomes possible. -- Bazsi
Balazs Scheidler wrote:
Checked the source, it is taken from the global option, and this patch allows it to set it for TCP/UDP targets (though the latter will never drop messages):
Index: cfg-grammar.y =================================================================== RCS file: /var/cvs/syslog-ng/syslog-ng/src/cfg-grammar.y,v retrieving revision 1.58.4.4 diff -u -r1.58.4.4 cfg-grammar.y --- cfg-grammar.y 6 May 2004 08:57:52 -0000 1.58.4.4 +++ cfg-grammar.y 17 Sep 2004 09:18:38 -0000 @@ -497,7 +497,8 @@ ;
dest_afinet_option - : KW_LOCALIP '(' string ')' { afinet_dest_set_localip(last_dest_driver, $3); free($3); } + : dest_item_option + | KW_LOCALIP '(' string ')' { afinet_dest_set_localip(last_dest_driver, $3); free($3); } | KW_LOCALPORT '(' NUMBER ')' { afinet_dest_set_localport(last_dest_driver, $3, NULL, NULL); } | KW_PORT '(' NUMBER ')' { afinet_dest_set_destport(last_dest_driver, $3, NULL, NULL); } | KW_DESTPORT '(' NUMBER ')' { afinet_dest_set_destport(last_dest_driver, $3, NULL, NULL); }
After applying the patch, rerun make (you fill need bison installed to rebuild)
Thanks for the fantastic response and patch. Knowing it works at the global level with log_fifo_size is great for my purposes; I just needed to know when message would be lost and increasing it globally should be sufficient for me. I've used a package on FreeBSD (rather than compiled myself as I have on Solaris) so I'll need to get the source and apply. Once I've done this I'll confirm it is OK. As an aside I assume when you say "TCP/UDP targets (though the latter will never drop messages)" you are referring to the the fact that the server sending UDP messages to another server will send them regardless of whether that second server is listening (because it's UDP) and so is not recorded as a dropped message? Thanks again, Jim Mozley
On Mon, 2004-09-20 at 11:21, Jim Mozley wrote:
As an aside I assume when you say "TCP/UDP targets (though the latter will never drop messages)" you are referring to the the fact that the server sending UDP messages to another server will send them regardless of whether that second server is listening (because it's UDP) and so is not recorded as a dropped message?
exactly. -- Bazsi
participants (2)
-
Balazs Scheidler
-
Jim Mozley