<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
--></style>
</head>
<body class='hmmessage'>
pzolee,<BR>
<BR>
The client happens to be a custom application, so I don't have a client config, and flow control doesn't really apply on the client side. I was able to setup a test environment and recreated the problem. The message immediately before the disconnect message is the following:<BR>
<BR>
<47>1 2010-01-11T14:36:40.239-06:00 server-04 syslog-ng 30082 - [meta sequenceId="122761"] debug Destination queue full, dropping message; queue_len='1000', mem_fifo_size='1000'<BR>
<BR>
I am guessing if I don't have flow control on the client side that I need to play with the numbers to ensure that none of the buffers ever get filled up. Is this correct?<BR>
<BR>
Thanks again,<BR>
Jim<BR>
<BR> <BR>
<HR id=stopSpelling>
Date: Thu, 7 Jan 2010 21:11:15 +0100<BR>From: pzolee@balabit.hu<BR>To: syslog-ng@lists.balabit.hu<BR>CC: jim_pirman@hotmail.com<BR>Subject: Re: [syslog-ng] Broken TCP connection<BR><BR>2010.01.07. 17:53 keltezéssel, James Pirman írta:
<BLOCKQUOTE cite=mid:SNT102-W194F9A751156F9AB19161EF7710@phx.gbl>
<STYLE>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Verdana;}
</STYLE>
Yes, that is correct. The 127.0.0.1 destination is actually my own application. <BR></BLOCKQUOTE>Ok<BR>
<BLOCKQUOTE cite=mid:SNT102-W194F9A751156F9AB19161EF7710@phx.gbl> <BR>I just noticed today that the problem seems to be happening when the amount of traffic increases. Right now I am testing with log_fetch_limit increased from 100 to 1000, and I added log_fifo_size globally and set it to 50000. I also decreased my flush timeout from 100 to 10. This appears to be helping and I haven't dropped a connection since. Does this seem like the correct approach?<BR></BLOCKQUOTE>I think, this is just a game with numbers but not the real reason for this behaviour. If you have problem with large traffic, just write the "flags(flow-control)" field into the right destination of your client config.<BR>Answer me that I asked of you, please (client config and debug log)<BR>
<BLOCKQUOTE cite=mid:SNT102-W194F9A751156F9AB19161EF7710@phx.gbl> <BR>Thanks,<BR>Jim<BR> <BR>
<HR id=ecxstopSpelling>
Date: Thu, 7 Jan 2010 17:38:41 +0100<BR>From: <A class=ecxmoz-txt-link-abbreviated href="mailto:pzolee@balabit.hu">pzolee@balabit.hu</A><BR>To: <A class=ecxmoz-txt-link-abbreviated href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</A>; <A class=ecxmoz-txt-link-abbreviated href="mailto:jim_pirman@hotmail.com">jim_pirman@hotmail.com</A><BR>Subject: Re: [syslog-ng] Broken TCP connection<BR><BR>Hi,<BR><BR>If I understand you correctly, you have three client/servers, don't you?<BR>client(.218) -> relay server(.198) -> local server on relay server (127.0.0.1)<BR><BR>and the problem is that sometimes your relay server drops the connection of client.<BR><BR><BR>James Pirman írta:
<BLOCKQUOTE cite=mid:SNT102-W3130D37832D34D2AB4B576F7710@phx.gbl>
<STYLE>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Verdana;}
</STYLE>
Is there anyone that can help with this? Is there any more information that I need to provide in order for me to get help? I've been dealing with for weeks and am starting to think the only solution is to write my own server.<BR> <BR>
<HR id=ecxecxstopSpelling>
From: <A class=ecxecxmoz-txt-link-abbreviated href="mailto:jim_pirman@hotmail.com">jim_pirman@hotmail.com</A><BR>To: <A class=ecxecxmoz-txt-link-abbreviated href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</A><BR>Date: Tue, 5 Jan 2010 11:22:36 -0600<BR>Subject: [syslog-ng] Broken TCP connection<BR><BR>
<STYLE>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Verdana;}
</STYLE>
I am currently having an issue with syslog-ng 3.0.4 where my TCP connection between my client and server is lost throughout the day. By looking at the pcap file from tcpdump I can tell that the TCP connection reset was initiated by the syslog-ng server. The only information that was initially in the log file regarding this disconnection was the following 2 lines:<BR> <BR><SPAN lang=EN><45>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733719"] notice Syslog connection closed; fd='9', client='AF_INET(192.168.27.218:46326)', local='AF_INET(192.168.27.198:20514)'<BR></SPAN></BLOCKQUOTE>Can you show me the previous few lines before this log message?<BR>Because if syslog-ng drops the connection usually sends log message about the reason of this behaviour, like this:<BR><BR>2010-01-07T17:24:48+01:00 syslog-ng err Invalid frame header; header=''<BR>2010-01-07T17:24:48+01:00 syslog-ng notice Syslog connection closed; fd='10', client='AF_INET(10.100.20.1:33251)', local='AF_INET(10.30.0.32:20514)'<BR><BR><BR>Your client config can also be useful, the problem may be on client side. Can you show me the debug log of your client when the connection lost?<BR>
<BLOCKQUOTE cite=mid:SNT102-W3130D37832D34D2AB4B576F7710@phx.gbl><SPAN lang=EN> <BR>and <BR> <BR><SPAN lang=EN><46>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733720"] info Closing log transport fd; fd='9'<BR> <BR> <BR>In order to get more information, I set the following flags in init.d: "-v -d -t".<BR> <BR>This did not give me any more information about the TCP disconnect, however I did notice that a lot of my normal messages were preceeded by the following text:<BR> <BR><SPAN lang=EN><47>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng 29213 - [meta sequenceId="2733718"] debug Incoming log entry; line=<BR> <BR>A normal log message then follows the '=' sign. <BR> <BR>A decent percentage of my messages are preceeded by this throughout the day, but just before the disconnect it appears that all of my messages from server-db-01 are preceeded by the debug line. Any ideas as to what could be going on? I have included my config file below if that helps.<BR> <BR>Any assistance would be greatly appreciated.<BR>-Jim<BR> <BR><SPAN lang=EN>@version: 3.0<BR>#Default configuration file for syslog-ng.<BR>#<BR># For a description of syslog-ng configuration file directives, please read<BR># the syslog-ng Administrator's guide at:<BR>#<BR># <A class=ecxecxmoz-txt-link-freetext href="http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html">http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html</A><BR>#<BR>options {<BR>keep_hostname(yes);<BR>keep_timestamp(yes);<BR>frac_digits(3);<BR>};<BR>source all {<BR>internal();<BR>syslog(ip("192.168.27.198") port(20514) transport("tcp") log_fetch_limit(100));<BR>};<BR>destination allclientsfile {<BR>file("/data/local/Logs/server-$YEAR-$MONTH-$DAY.log"<BR>flags(syslog-protocol)<BR>flush_timeout(100)<BR>create_dirs(yes)<BR>dir_owner(jpirman)<BR>dir_group(jpirman)<BR>owner(jpirman)<BR>group(jpirman)<BR>template("$PRIORITY $MESSAGE")<BR>);<BR>};<BR>destination msgserver {<BR>udp("127.0.0.1" port(20515)<BR>flush_timeout(100) <BR>template("$ISODATE $PROGRAM $PRIORITY $MESSAGE\n"));<BR>};<BR>log { source(all); destination(allclientsfile); destination(msgserver);};<BR></SPAN> <BR> <BR></SPAN></SPAN></SPAN><BR>
<HR>
Hotmail: Powerful Free email with security by Microsoft. <A href="http://clk.atdmt.com/GBL/go/171222986/direct/01/">Get it now.</A> <BR>
<HR>
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. <A href="http://clk.atdmt.com/GBL/go/196390709/direct/01/">Sign up now.</A> <PRE><HR width="90%" SIZE=4>
______________________________________________________________________________
Member info: <A class=ecxecxmoz-txt-link-freetext href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</A>
Documentation: <A class=ecxecxmoz-txt-link-freetext href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</A>
FAQ: <A class=ecxecxmoz-txt-link-freetext href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</A>
</PRE></BLOCKQUOTE><BR><BR><PRE class=ecxecxmoz-signature>--
pzolee
</PRE><BR>
<HR>
Your E-mail and More On-the-Go. Get Windows Live Hotmail Free. <A href="http://clk.atdmt.com/GBL/go/196390709/direct/01/">Sign up now.</A> <PRE><FIELDSET class=ecxmimeAttachmentHeader></FIELDSET>
______________________________________________________________________________
Member info: <A class=ecxmoz-txt-link-freetext href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</A>
Documentation: <A class=ecxmoz-txt-link-freetext href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</A>
FAQ: <A class=ecxmoz-txt-link-freetext href="http://www.campin.net/syslog-ng/faq.html">http://www.campin.net/syslog-ng/faq.html</A>
</PRE></BLOCKQUOTE><BR><BR>
<DIV class=ecxmoz-signature>-- <BR>pzolee</DIV> <br /><hr />Hotmail: Free, trusted and rich email service. <a href='http://clk.atdmt.com/GBL/go/196390708/direct/01/' target='_new'>Get it now.</a></body>
</html>