[syslog-ng] Broken TCP connection

Mon Jan 11 21:49:07 CET 2010

pzolee,

The client happens to be a custom application, so I don't have a client config, and flow control doesn't really apply on the client side.  I was able to setup a test environment and recreated the problem.  The message immediately before the disconnect message is the following:

<47>1 2010-01-11T14:36:40.239-06:00 server-04 syslog-ng 30082 - [meta sequenceId="122761"] debug Destination queue full, dropping message; queue_len='1000', mem_fifo_size='1000'

I am guessing if I don't have flow control on the client side that I need to play with the numbers to ensure that none of the buffers ever get filled up.  Is this correct?

Thanks again,

Jim

Date: Thu, 7 Jan 2010 21:11:15 +0100
From: pzolee at balabit.hu
To: syslog-ng at lists.balabit.hu
CC: jim_pirman at hotmail.com
Subject: Re: [syslog-ng] Broken TCP connection

2010.01.07. 17:53 keltezéssel, James Pirman írta: 

Yes, that is correct.  The 127.0.0.1 destination is actually my own application.  
Ok

I just noticed today that the problem seems to be happening when the amount of traffic increases.  Right now I am testing with log_fetch_limit increased from 100 to 1000, and I added log_fifo_size globally and set it to 50000.  I also decreased my flush timeout from 100 to 10.  This appears to be helping and I haven't dropped a connection since.  Does this seem like the correct approach?
I think, this is just a game with numbers but not the real reason for this behaviour. If you have problem with large traffic, just write the "flags(flow-control)" field into the right destination of your client config.
Answer me that I asked of you, please (client config and debug log)

Thanks,
Jim

Date: Thu, 7 Jan 2010 17:38:41 +0100
From: pzolee at balabit.hu
To: syslog-ng at lists.balabit.hu; jim_pirman at hotmail.com
Subject: Re: [syslog-ng] Broken TCP connection

Hi,

If I understand you correctly, you have three client/servers, don't you?
client(.218) -> relay server(.198) -> local server on relay server (127.0.0.1)

and the problem is that sometimes your relay server drops the connection of client.

James Pirman írta: 

Is there anyone that can help with this?  Is there any more information that I need to provide in order for me to get help? I've been dealing with for weeks and am starting to think the only solution is to write my own server.

From: jim_pirman at hotmail.com
To: syslog-ng at lists.balabit.hu
Date: Tue, 5 Jan 2010 11:22:36 -0600
Subject: [syslog-ng] Broken TCP connection

I am currently having an issue with syslog-ng 3.0.4 where my TCP connection between my client and server is lost throughout the day.  By looking at the pcap file from tcpdump I can tell that the TCP connection reset was initiated by the syslog-ng server.  The only information that was initially in the log file regarding this disconnection was the following 2 lines:

<45>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733719"] notice Syslog connection closed; fd='9', client='AF_INET(192.168.27.218:46326)', local='AF_INET(192.168.27.198:20514)'
Can you show me the previous few lines before this log message?
Because if syslog-ng drops the connection usually sends log message about the reason of this behaviour, like this:

2010-01-07T17:24:48+01:00 syslog-ng err Invalid frame header; header=''
2010-01-07T17:24:48+01:00 syslog-ng notice Syslog connection closed; fd='10', client='AF_INET(10.100.20.1:33251)', local='AF_INET(10.30.0.32:20514)'

Your client config can also be useful, the problem may be on client side. Can you show me the debug log of your client when the connection lost?

and 

<46>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733720"] info Closing log transport fd; fd='9'

In order to get more information, I set the following flags in init.d: "-v -d -t".

This did not give me any more information about the TCP disconnect, however I did notice that a lot of my normal messages were preceeded by the following text:

<47>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733718"] debug Incoming log entry; line=

A normal log message then follows the '=' sign. 

A decent percentage of my messages are preceeded by this throughout the day, but just before the disconnect it appears that all of my messages from server-db-01 are preceeded by the debug line.  Any ideas as to what could be going on?  I have included my config file below if that helps.

Any assistance would be greatly appreciated.
-Jim

@version: 3.0
#Default configuration file for syslog-ng.
#
# For a description of syslog-ng configuration file directives, please read
# the syslog-ng Administrator's guide at:
#
# http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html
#
options {
keep_hostname(yes);
keep_timestamp(yes);
frac_digits(3);
};
source all {
internal();
syslog(ip("192.168.27.198") port(20514) transport("tcp") log_fetch_limit(100));
};
destination allclientsfile {
file("/data/local/Logs/server-$YEAR-$MONTH-$DAY.log"
flags(syslog-protocol)
flush_timeout(100)
create_dirs(yes)
dir_owner(jpirman)
dir_group(jpirman)
owner(jpirman)
group(jpirman)
template("$PRIORITY $MESSAGE")
);
};
destination msgserver {
udp("127.0.0.1" port(20515)
flush_timeout(100) 
template("$ISODATE $PROGRAM $PRIORITY $MESSAGE\n"));
};
log { source(all); destination(allclientsfile); destination(msgserver);};

Hotmail: Powerful Free email with security by Microsoft. Get it now. 

Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. Sign up now. 
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

-- 
pzolee

Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. Sign up now. 
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

-- 
pzolee 		 	   		  
_________________________________________________________________
Hotmail: Free, trusted and rich email service.
http://clk.atdmt.com/GBL/go/196390708/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100111/f71fd6d0/attachment.htm 