[syslog-ng] Broken TCP connection
Zoltán Pallagi
pzolee at balabit.hu
Tue Jan 12 21:17:34 CET 2010
Hi,
Yes, it is. You need to increase log_fetch_limit() and log_fifo_size()
in this case. Keep your mind the log_fifo_size must be larger than the
log_fetch_limit.
I don't suggest reducing flush_timeout but you should increase both
flush_timeout and flush_lines. Because this message (dropping message)
means your client send messages too fast and the syslog-ng cannot be
able to write them in time.
Back to the original problem, I don't have much idea the reason for
disconnecting. Can you send me the tcpdump file? I suspect the problem
may be here in relation to high traffic.
2010.01.11. 21:49 keltezéssel, James Pirman írta:
> pzolee,
>
> The client happens to be a custom application, so I don't have a
> client config, and flow control doesn't really apply on the client
> side. I was able to setup a test environment and recreated the
> problem. The message immediately before the disconnect message is the
> following:
>
> <47>1 2010-01-11T14:36:40.239-06:00 server-04 syslog-ng 30082 - [meta
> sequenceId="122761"] debug Destination queue full, dropping message;
> queue_len='1000', mem_fifo_size='1000'
>
> I am guessing if I don't have flow control on the client side that I
> need to play with the numbers to ensure that none of the buffers ever
> get filled up. Is this correct?
>
> Thanks again,
> Jim
>
>
> ------------------------------------------------------------------------
> Date: Thu, 7 Jan 2010 21:11:15 +0100
> From: pzolee at balabit.hu
> To: syslog-ng at lists.balabit.hu
> CC: jim_pirman at hotmail.com
> Subject: Re: [syslog-ng] Broken TCP connection
>
> 2010.01.07. 17:53 keltezéssel, James Pirman írta:
>
> Yes, that is correct. The 127.0.0.1 destination is actually my
> own application.
>
> Ok
>
>
> I just noticed today that the problem seems to be happening when
> the amount of traffic increases. Right now I am testing with
> log_fetch_limit increased from 100 to 1000, and I added
> log_fifo_size globally and set it to 50000. I also decreased my
> flush timeout from 100 to 10. This appears to be helping and I
> haven't dropped a connection since. Does this seem like the
> correct approach?
>
> I think, this is just a game with numbers but not the real reason for
> this behaviour. If you have problem with large traffic, just write the
> "flags(flow-control)" field into the right destination of your client
> config.
> Answer me that I asked of you, please (client config and debug log)
>
>
> Thanks,
> Jim
>
> ------------------------------------------------------------------------
> Date: Thu, 7 Jan 2010 17:38:41 +0100
> From: pzolee at balabit.hu <mailto:pzolee at balabit.hu>
> To: syslog-ng at lists.balabit.hu
> <mailto:syslog-ng at lists.balabit.hu>; jim_pirman at hotmail.com
> <mailto:jim_pirman at hotmail.com>
> Subject: Re: [syslog-ng] Broken TCP connection
>
> Hi,
>
> If I understand you correctly, you have three client/servers,
> don't you?
> client(.218) -> relay server(.198) -> local server on relay server
> (127.0.0.1)
>
> and the problem is that sometimes your relay server drops the
> connection of client.
>
>
> James Pirman írta:
>
> Is there anyone that can help with this? Is there any more
> information that I need to provide in order for me to get
> help? I've been dealing with for weeks and am starting to
> think the only solution is to write my own server.
>
> ------------------------------------------------------------------------
> From: jim_pirman at hotmail.com <mailto:jim_pirman at hotmail.com>
> To: syslog-ng at lists.balabit.hu <mailto:syslog-ng at lists.balabit.hu>
> Date: Tue, 5 Jan 2010 11:22:36 -0600
> Subject: [syslog-ng] Broken TCP connection
>
> I am currently having an issue with syslog-ng 3.0.4 where my
> TCP connection between my client and server is lost throughout
> the day. By looking at the pcap file from tcpdump I can tell
> that the TCP connection reset was initiated by the syslog-ng
> server. The only information that was initially in the log
> file regarding this disconnection was the following 2 lines:
>
> <45>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng
> 29213 - [meta sequenceId="2733719"] notice Syslog connection
> closed; fd='9', client='AF_INET(192.168.27.218:46326)',
> local='AF_INET(192.168.27.198:20514)'
>
> Can you show me the previous few lines before this log message?
> Because if syslog-ng drops the connection usually sends log
> message about the reason of this behaviour, like this:
>
> 2010-01-07T17:24:48+01:00 syslog-ng err Invalid frame header;
> header=''
> 2010-01-07T17:24:48+01:00 syslog-ng notice Syslog connection
> closed; fd='10', client='AF_INET(10.100.20.1:33251)',
> local='AF_INET(10.30.0.32:20514)'
>
>
> Your client config can also be useful, the problem may be on
> client side. Can you show me the debug log of your client when the
> connection lost?
>
>
> and
>
> <46>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng
> 29213 - [meta sequenceId="2733720"] info Closing log transport
> fd; fd='9'
>
>
> In order to get more information, I set the following flags in
> init.d: "-v -d -t".
>
> This did not give me any more information about the TCP
> disconnect, however I did notice that a lot of my normal
> messages were preceeded by the following text:
>
> <47>1 2010-01-05T10:29:32.323-06:00 server-db-01 syslog-ng
> 29213 - [meta sequenceId="2733718"] debug Incoming log entry;
> line=
>
> A normal log message then follows the '=' sign.
>
> A decent percentage of my messages are preceeded by this
> throughout the day, but just before the disconnect it appears
> that all of my messages from server-db-01 are preceeded by the
> debug line. Any ideas as to what could be going on? I have
> included my config file below if that helps.
>
> Any assistance would be greatly appreciated.
> -Jim
>
> @version: 3.0
> #Default configuration file for syslog-ng.
> #
> # For a description of syslog-ng configuration file
> directives, please read
> # the syslog-ng Administrator's guide at:
> #
> #
> http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html
> #
> options {
> keep_hostname(yes);
> keep_timestamp(yes);
> frac_digits(3);
> };
> source all {
> internal();
> syslog(ip("192.168.27.198") port(20514) transport("tcp")
> log_fetch_limit(100));
> };
> destination allclientsfile {
> file("/data/local/Logs/server-$YEAR-$MONTH-$DAY.log"
> flags(syslog-protocol)
> flush_timeout(100)
> create_dirs(yes)
> dir_owner(jpirman)
> dir_group(jpirman)
> owner(jpirman)
> group(jpirman)
> template("$PRIORITY $MESSAGE")
> );
> };
> destination msgserver {
> udp("127.0.0.1" port(20515)
> flush_timeout(100)
> template("$ISODATE $PROGRAM $PRIORITY $MESSAGE\n"));
> };
> log { source(all); destination(allclientsfile);
> destination(msgserver);};
>
>
>
> ------------------------------------------------------------------------
> Hotmail: Powerful Free email with security by Microsoft. Get
> it now. <http://clk.atdmt.com/GBL/go/171222986/direct/01/>
> ------------------------------------------------------------------------
> Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
> Sign up now. <http://clk.atdmt.com/GBL/go/196390709/direct/01/>
>
> ------------------------------------------------------------------------
>
> ______________________________________________________________________________
> Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ:http://www.campin.net/syslog-ng/faq.html
>
>
>
>
>
> --
> pzolee
>
>
>
> ------------------------------------------------------------------------
> Your E-mail and More On-the-Go. Get Windows Live Hotmail Free.
> Sign up now. <http://clk.atdmt.com/GBL/go/196390709/direct/01/>
>
>
> ______________________________________________________________________________
> Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ:http://www.campin.net/syslog-ng/faq.html
>
>
>
>
>
> --
> pzolee
>
> ------------------------------------------------------------------------
> Hotmail: Free, trusted and rich email service. Get it now.
> <http://clk.atdmt.com/GBL/go/196390708/direct/01/>
>
>
> ______________________________________________________________________________
> Member info:https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ:http://www.campin.net/syslog-ng/faq.html
>
>
--
pzolee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100112/c284148c/attachment.htm
More information about the syslog-ng
mailing list