syslog-ng not reconnecting when using tcp
Hi, we use syslog-ng on one of our nodes as client and rsyslogd as server. We have configured tcp to connect to server. What we see is when the reboot the complete cluster (has node where the server runs and other node where the client runs,) only the server node is rebooted. The client node where syslog-ng runs is not rebooted. The issue is after we reboot the syslog-ng connection is not re-established and hence we are not getting any logs on the server. After going through some of the posts related to the problem and the admin guide, it mentions about time_reopen() value to solve this issue. In my case, i have configured time_reopen(40). So it means at 40s interval the tcp connection is tired to be re-established. But i see that this is not happening. Can you please help me resolve this issue. Also can you let me know how many times and at what interval this connection re-establishment is tried. Below are some of the logs that could be helpful. syslog-ng.conf: cat /etc/syslog-ng/syslog-ng.conf # syslog-ng configuration file. # # This should behave pretty much like the original syslog on RedHat. But # it could be configured a lot smarter. # # See syslog-ng(8) and syslog-ng.conf(5) for more information. # # 20000925 gb@sysfive.com # # Updated by Frank Crawford (<Frank.Crawford@ac3.com.au>) - 10 Aug 2002 # - for Red Hat 7.3 # - totally do away with klogd # - add message "kernel:" as is done with klogd. # # Updated by Frank Crawford (<Frank.Crawford@ac3.com.au>) - 22 Aug 2002 # - use the log_prefix option as per Balazs Scheidler's email # # Updated by Jose Pedro Oliveira (<jpo at di.uminho.pt>) - 05 Apr 2003 # - corrected filters 'f_filter2' and 'f_filter6' # these filters were only allowing messages of one specific # priority level; they should be allowing messages from that # priority and upper levels. # # Updated by Jose Pedro Oliveira (<jpo at di.uminho.pt>) - 25 Jan 2005 # - Don't sync the d_mail destination # # Updated by Jose Pedro Oliveira (<jpo at di.uminho.pt>) - 01 Feb 2005 # - /proc/kmsg is a file not a pipe. # ( https://lists.balabit.hu/pipermail/syslog-ng/2005-February/006963.html) # options { sync (0); time_reopen (40); log_fifo_size (1000); stats(86400); long_hostnames (off); use_dns (no); use_fqdn (no); create_dirs (no); keep_hostname (yes); }; source s_sys { file ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); # udp(ip(0.0.0.0) port(514)); }; destination d_cons { file("/dev/console"); }; #destination d_mesg { file("/var/log/messages"); }; #destination d_auth { file("/var/log/secure"); }; destination d_mail { file("/var/log/maillog" sync(10)); }; destination d_spol { file("/var/log/spooler"); }; destination d_boot { file("/var/log/boot.log"); }; destination d_cron { file("/var/log/cron"); }; destination d_mlal { usertty("*"); }; destination tcp-to-master { tcp("169.254.1.82" localip("169.254.1.66") localport(601) port(601) ); }; #filter f_filter1 { facility(kern); }; filter f_filter2 { level(info..emerg) and not facility(mail,authpriv,cron) and not match("Connection broken to AF_INET") and not match("Error connecting to remote host AF_INET"); }; filter f_filter3 { facility(authpriv); }; filter f_filter4 { facility(mail); }; filter f_filter5 { level(emerg); }; filter f_filter6 { facility(uucp) or (facility(news) and level(crit..emerg)); }; filter f_filter7 { facility(local7); }; filter f_filter8 { facility(cron); }; log { source(s_sys); filter(f_filter2); destination(tcp-to-master); }; #log { source(s_sys); filter(f_filter1); destination(d_cons); }; # Redirecting the logs to /var/log/ of CLA, to avoid log files filling up AHUB3-A file system #log { source(s_sys); filter(f_filter2); destination(d_mesg); }; #log { source(s_sys); filter(f_filter3); destination(d_auth); }; log { source(s_sys); filter(f_filter4); destination(d_mail); }; log { source(s_sys); filter(f_filter5); destination(d_mlal); }; log { source(s_sys); filter(f_filter6); destination(d_spol); }; log { source(s_sys); filter(f_filter7); destination(d_boot); }; log { source(s_sys); filter(f_filter8); destination(d_cron); }; netstat output when the cluster was rebooted: Transition from ESTABLISHED to CLOSED_WAIT tcp 0 0 169.254.1.67:601 169.254.1.82:601 ESTABLISHED 3771/syslog-ng 10.34.37.082224000 tcp 1 0 169.254.1.67:601 169.254.1.82:601 CLOSE_WAIT 3771/syslog-ng CLOSED_WAIT to LAST_ACK tcp 1 0 169.254.1.67:601 169.254.1.82:601 CLOSE_WAIT 3771/syslog-ng 10.34.43.555017000 tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.34.43.644520000 LAST_ACK to CLOSED: tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.36.29.754533000 tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.36.29.801733000 tcp 1 1 169.254.1.67:601 169.254.1.82:601 LAST_ACK - 10.36.29.846228000 10.36.29.892427000 10.36.29.931181000 SYN_SENT: 10.36.43.192808000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng 10.36.43.238811000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng 10.36.43.279561000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng continues... 10.36.46.180840000 tcp 0 1 169.254.1.67:601 169.254.1.82:601 SYN_SENT 3771/syslog-ng 10.36.46.225891000 10.36.46.275090000 10.36.46.318485000 10.36.46.359169000 The server node comes up and it start rsyslogd after some more time. But when this starts, there is no syn packets received by rsyslogd and hence the connection is not established. Can you please let me know where exactly 40s(configured time_reopen value) coming into picture here in the netstat output? syslog-ng version: syslog-ng 1.6.12 ----------------------------------------------------------------------------------------------------------------------------------------------------- I got a reply from one of the balabit guys that i have to use version 3.4. I am currently trying to use it. But i still have some questions below. 1. How many number of times the connection is retried. 2. when exactly the connection retry happens (when will it get to know when it has to retry) 3. time_reopen(40), what exactly is 40seconds here. -- Regards, Prasad
prasad padiyar <prasad.padiyar@gmail.com> writes: [...]
syslog-ng version:
syslog-ng 1.6.12 [...]
I got a reply from one of the balabit guys that i have to use version 3.4.
Indeed, 1.6 is ancient.
1. How many number of times the connection is retried.
It is retried every time_reopen() seconds until it succeeds, or syslog-ng is stopped.
2. when exactly the connection retry happens (when will it get to know when it has to retry)
The retry happens after syslog-ng disconnected (and syslog-ng disconnects when it notices a network error, usually during a write() or a poll()), after time_reopen() seconds have passed.
3. time_reopen(40), what exactly is 40seconds here.
The 40 seconds here is the time spent with the destination suspended after an error, before trying to reconnect again. Hope that answers your questions! -- |8]
Thanks for the kind reply.
1. How many number of times the connection is retried.
It is retried every time_reopen() seconds until it succeeds, or syslog-ng is stopped. But somehow I do not see this happening in my case. Is there some configuration in syslog-ng.conf that does this (retrying until it succeeds or syslog-ng is stopped) ? On Wed, Apr 2, 2014 at 5:49 PM, Gergely Nagy <algernon@balabit.hu> wrote:
prasad padiyar <prasad.padiyar@gmail.com> writes:
[...]
syslog-ng version:
syslog-ng 1.6.12 [...]
I got a reply from one of the balabit guys that i have to use version 3.4.
Indeed, 1.6 is ancient.
1. How many number of times the connection is retried.
It is retried every time_reopen() seconds until it succeeds, or syslog-ng is stopped.
2. when exactly the connection retry happens (when will it get to know when it has to retry)
The retry happens after syslog-ng disconnected (and syslog-ng disconnects when it notices a network error, usually during a write() or a poll()), after time_reopen() seconds have passed.
3. time_reopen(40), what exactly is 40seconds here.
The 40 seconds here is the time spent with the destination suspended after an error, before trying to reconnect again.
Hope that answers your questions!
-- |8]
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Regards, Prasad
participants (2)
-
Gergely Nagy
-
prasad padiyar