Hi, Yes, it is. You need to increase log_fetch_limit() and log_fifo_size() in this case. Keep your mind the log_fifo_size must be larger than the log_fetch_limit. I don't suggest reducing flush_timeout but you should increase both flush_timeout and flush_lines. Because this message (dropping message) means your client send messages too fast and the syslog-ng cannot be able to write them in time. Back to the original problem, I don't have much idea the reason for disconnecting. Can you send me the tcpdump file? I suspect the problem may be here in relation to high traffic. 2010.01.11. 21:49 keltezéssel, James Pirman írta:
pzolee,
The client happens to be a custom application, so I don't have a client config, and flow control doesn't really apply on the client side. I was able to setup a test environment and recreated the problem. The message immediately before the disconnect message is the following:
<47>1 2010-01-11T14:36:40.239-06:00 server-04 syslog-ng 30082 - [meta sequenceId="122761"] debug Destination queue full, dropping message; queue_len='1000', mem_fifo_size='1000'
I am guessing if I don't have flow control on the client side that I need to play with the numbers to ensure that none of the buffers ever get filled up. Is this correct?
Thanks again, Jim
------------------------------------------------------------------------ Date: Thu, 7 Jan 2010 21:11:15 +0100 From: pzolee@balabit.hu To: syslog-ng@lists.balabit.hu CC: jim_pirman@hotmail.com Subject: Re: [syslog-ng] Broken TCP connection
2010.01.07. 17:53 keltezéssel, James Pirman írta:
Yes, that is correct. The 127.0.0.1 destination is actually my own application.
Ok
I just noticed today that the problem seems to be happening when the amount of traffic increases. Right now I am testing with log_fetch_limit increased from 100 to 1000, and I added log_fifo_size globally and set it to 50000. I also decreased my flush timeout from 100 to 10. This appears to be helping and I haven't dropped a connection since. Does this seem like the correct approach?
I think, this is just a game with numbers but not the real reason for this behaviour. If you have problem with large traffic, just write the "flags(flow-control)" field into the right destination of your client config. Answer me that I asked of you, please (client config and debug log)
Thanks, Jim
------------------------------------------------------------------------ Date: Thu, 7 Jan 2010 17:38:41 +0100 From: pzolee@balabit.hu <mailto:pzolee@balabit.hu> To: syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu>; jim_pirman@hotmail.com <mailto:jim_pirman@hotmail.com> Subject: Re: [syslog-ng] Broken TCP connection
Hi,
If I understand you correctly, you have three client/servers, don't you? client(.218) -> relay server(.198) -> local server on relay server (127.0.0.1)
and the problem is that sometimes your relay server drops the connection of client.
James Pirman írta:
Is there anyone that can help with this? Is there any more information that I need to provide in order for me to get help? I've been dealing with for weeks and am starting to think the only solution is to write my own server.
------------------------------------------------------------------------ From: jim_pirman@hotmail.com <mailto:jim_pirman@hotmail.com> To: syslog-ng@lists.balabit.hu <mailto:syslog-ng@lists.balabit.hu> Date: Tue, 5 Jan 2010 11:22:36 -0600 Subject: [syslog-ng] Broken TCP connection
I am currently having an issue with syslog-ng 3.0.4 where my TCP connection between my client and server is lost throughout the day. By looking at the pcap file from tcpdump I can tell that the TCP connection reset was initiated by the syslog-ng server. The only information that was initially in the log file regarding this disconnection was the following 2 lines:
<45>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733719"] notice Syslog connection closed; fd='9', client='AF_INET(192.168.27.218:46326)', local='AF_INET(192.168.27.198:20514)'
Can you show me the previous few lines before this log message? Because if syslog-ng drops the connection usually sends log message about the reason of this behaviour, like this:
2010-01-07T17:24:48+01:00 syslog-ng err Invalid frame header; header='' 2010-01-07T17:24:48+01:00 syslog-ng notice Syslog connection closed; fd='10', client='AF_INET(10.100.20.1:33251)', local='AF_INET(10.30.0.32:20514)'
Your client config can also be useful, the problem may be on client side. Can you show me the debug log of your client when the connection lost?
and
<46>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733720"] info Closing log transport fd; fd='9'
In order to get more information, I set the following flags in init.d: "-v -d -t".
This did not give me any more information about the TCP disconnect, however I did notice that a lot of my normal messages were preceeded by the following text:
<47>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733718"] debug Incoming log entry; line=
A normal log message then follows the '=' sign.
A decent percentage of my messages are preceeded by this throughout the day, but just before the disconnect it appears that all of my messages from server-db-01 are preceeded by the debug line. Any ideas as to what could be going on? I have included my config file below if that helps.
Any assistance would be greatly appreciated. -Jim
@version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html # options { keep_hostname(yes); keep_timestamp(yes); frac_digits(3); }; source all { internal(); syslog(ip("192.168.27.198") port(20514) transport("tcp") log_fetch_limit(100)); }; destination allclientsfile { file("/data/local/Logs/server-$YEAR-$MONTH-$DAY.log" flags(syslog-protocol) flush_timeout(100) create_dirs(yes) dir_owner(jpirman) dir_group(jpirman) owner(jpirman) group(jpirman) template("$PRIORITY $MESSAGE") ); }; destination msgserver { udp("127.0.0.1" port(20515) flush_timeout(100) template("$ISODATE $PROGRAM $PRIORITY $MESSAGE\n")); }; log { source(all); destination(allclientsfile); destination(msgserver);};
------------------------------------------------------------------------ Hotmail: Powerful Free email with security by Microsoft. Get it now. <http://clk.atdmt.com/GBL/go/171222986/direct/01/> ------------------------------------------------------------------------ Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. Sign up now. <http://clk.atdmt.com/GBL/go/196390709/direct/01/>
------------------------------------------------------------------------
______________________________________________________________________________ Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation:http://www.balabit.com/support/documentation/?product=syslog-ng FAQ:http://www.campin.net/syslog-ng/faq.html
-- pzolee
------------------------------------------------------------------------ Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. Sign up now. <http://clk.atdmt.com/GBL/go/196390709/direct/01/>
______________________________________________________________________________ Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation:http://www.balabit.com/support/documentation/?product=syslog-ng FAQ:http://www.campin.net/syslog-ng/faq.html
-- pzolee
------------------------------------------------------------------------ Hotmail: Free, trusted and rich email service. Get it now. <http://clk.atdmt.com/GBL/go/196390708/direct/01/>
______________________________________________________________________________ Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation:http://www.balabit.com/support/documentation/?product=syslog-ng FAQ:http://www.campin.net/syslog-ng/faq.html
-- pzolee