[syslog-ng] Broken TCP connection

Thu Jan 7 17:53:10 CET 2010

Yes, that is correct.  The 127.0.0.1 destination is actually my own application.  

I just noticed today that the problem seems to be happening when the amount of traffic increases.  Right now I am testing with log_fetch_limit increased from 100 to 1000, and I added log_fifo_size globally and set it to 50000.  I also decreased my flush timeout from 100 to 10.  This appears to be helping and I haven't dropped a connection since.  Does this seem like the correct approach?

Thanks,

Jim

Date: Thu, 7 Jan 2010 17:38:41 +0100
From: pzolee at balabit.hu
To: syslog-ng at lists.balabit.hu; jim_pirman at hotmail.com
Subject: Re: [syslog-ng] Broken TCP connection

Hi,

If I understand you correctly, you have three client/servers, don't you?
client(.218) -> relay server(.198) -> local server on relay server (127.0.0.1)

and the problem is that sometimes your relay server drops the connection of client.

James Pirman írta: 

Is there anyone that can help with this?  Is there any more information that I need to provide in order for me to get help? I've been dealing with for weeks and am starting to think the only solution is to write my own server.

From: jim_pirman at hotmail.com
To: syslog-ng at lists.balabit.hu
Date: Tue, 5 Jan 2010 11:22:36 -0600
Subject: [syslog-ng] Broken TCP connection

I am currently having an issue with syslog-ng 3.0.4 where my TCP connection between my client and server is lost throughout the day.  By looking at the pcap file from tcpdump I can tell that the TCP connection reset was initiated by the syslog-ng server.  The only information that was initially in the log file regarding this disconnection was the following 2 lines:

<45>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733719"] notice Syslog connection closed; fd='9', client='AF_INET(192.168.27.218:46326)', local='AF_INET(192.168.27.198:20514)'
Can you show me the previous few lines before this log message?
Because if syslog-ng drops the connection usually sends log message about the reason of this behaviour, like this:

2010-01-07T17:24:48+01:00 syslog-ng err Invalid frame header; header=''
2010-01-07T17:24:48+01:00 syslog-ng notice Syslog connection closed; fd='10', client='AF_INET(10.100.20.1:33251)', local='AF_INET(10.30.0.32:20514)'

Your client config can also be useful, the problem may be on client side. Can you show me the debug log of your client when the connection lost?

and 

<46>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733720"] info Closing log transport fd; fd='9'

In order to get more information, I set the following flags in init.d: "-v -d -t".

This did not give me any more information about the TCP disconnect, however I did notice that a lot of my normal messages were preceeded by the following text:

<47>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733718"] debug Incoming log entry; line=

A normal log message then follows the '=' sign. 

A decent percentage of my messages are preceeded by this throughout the day, but just before the disconnect it appears that all of my messages from server-db-01 are preceeded by the debug line.  Any ideas as to what could be going on?  I have included my config file below if that helps.

Any assistance would be greatly appreciated.
-Jim

@version: 3.0
#Default configuration file for syslog-ng.
#
# For a description of syslog-ng configuration file directives, please read
# the syslog-ng Administrator's guide at:
#
# http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html
#
options {
keep_hostname(yes);
keep_timestamp(yes);
frac_digits(3);
};
source all {
internal();
syslog(ip("192.168.27.198") port(20514) transport("tcp") log_fetch_limit(100));
};
destination allclientsfile {
file("/data/local/Logs/server-$YEAR-$MONTH-$DAY.log"
flags(syslog-protocol)
flush_timeout(100)
create_dirs(yes)
dir_owner(jpirman)
dir_group(jpirman)
owner(jpirman)
group(jpirman)
template("$PRIORITY $MESSAGE")
);
};
destination msgserver {
udp("127.0.0.1" port(20515)
flush_timeout(100) 
template("$ISODATE $PROGRAM $PRIORITY $MESSAGE\n"));
};
log { source(all); destination(allclientsfile); destination(msgserver);};

Hotmail: Powerful Free email with security by Microsoft. Get it now. 

Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. Sign up now. 
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html

-- 
pzolee

_________________________________________________________________
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
http://clk.atdmt.com/GBL/go/196390709/direct/01/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100107/542d2a2e/attachment.htm 