syslog-ng and network problem
Hello guys I've installed two syslog-ng server instances behind a load balancer (LB) and configured N syslog-ng clients pointing to my virtual ip address. The clients doesn't send too much messages to the servers at this time. For some reason the LB closes the connections and when my client have to send a message to the server it realize that the connection was closed and it reopen a new one. The problem is that the messages even if the connection is stablished once again doesn't arrive to the server. I tryed also to enable the mark_freq(10) option but nothing changed. It starts to work if I restart the client or if I generate a lot of messages ( echo "MESSAGE" >> FILE.log ) I realize that the problem is in the network, but is someone have another idea to workaround this issue i'll be thankfull! Here is the config of one of my clients: @version: 3.0 #Default configuration file for syslog-ng. # # For a description of syslog-ng configuration file directives, please read # the syslog-ng Administrator's guide at: # # http://www.balabit.com/dl/html/syslog-ng-admin-guide_en.html/bk01-toc.html ######### # options options { time_reopen(10); keep_hostname(yes); log_msg_size(65535); flush_lines(0); }; ######### # sources source s_balabit { internal(); }; source s_source { file("FILE.log"); }; filter f_filter { match ("OUTGOING" value("MESSAGE")); }; ######### # destinations destination d_internal { file("/var/log/balabit.log"); }; destination d_network { tcp("vip" port(10001) tls( ca_dir("/etc/ssl/syslog-ng/ca.d")) log_fifo_size(4194304) ); }; ######### # logs log { source(s_source); filter(f_filter); destination(d_network);}; log { source(s_balabit); destination(d_internal); };
On p, 2010-06-11 at 12:01 +0200, Giovanni Mancuso wrote:
The problem is that the messages even if the connection is stablished once again doesn't arrive to the server.
Could you please send syslog-ng messages both on the client and the server side. I am interested in messages around the LB breaks the connection and it is re-establised. If I understand you correctly, the LB breaks the connection, than the client establish it again but no message is transfered. Right? Peter -- Höltzl Péter CISA, IT biztonsági tanácsadó holtzl.peter@balabit.hu +36 20 366 966 http://peter.blogs.balabit.hu/ BalaBit IT Security 1115 Budapest XI. Bártfai u. 54. Tel +36 1 371 0540 Fax +36 1 208 0875 Az üzenet és annak bármely csatolt anyaga bizalmas, jogi védelem alatt áll, a nyilvános közléstől védett. Az üzenetet kizárólag a címzett, illetve az általa meghatalmazottak használhatják fel. Ha Ön nem az üzenet címzettje, úgy kérjük, hogy telefonon, vagy e-mail-ben értesítse erről az üzenet küldőjét és törölje az üzenetet, valamint annak összes csatolt mellékletét a rendszeréből. Ha Ön nem az üzenet címzettje, abban az esetben tilos az üzenetet vagy annak bármely csatolt mellékletét lemásolnia, elmentenie, az üzenet tartalmát bárkivel közölnie vagy azzal visszaélnie.
HÖLTZL Péter wrote:
On p, 2010-06-11 at 12:01 +0200, Giovanni Mancuso wrote:
The problem is that the messages even if the connection is stablished once again doesn't arrive to the server.
Could you please send syslog-ng messages both on the client and the server side. I am interested in messages around the LB breaks the connection and it is re-establised. If I understand you correctly, the LB breaks the connection, than the client establish it again but no message is transfered. Right?
Peter
Right Peter. I paste a part of the log. Server: Jun 11 16:44:40 mmt-l-al36 syslog-ng[2318]: Syslog connection accepted; fd='15', client='AF_INET(10.35.37.168:46993)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:40 mmt-l-al36 syslog-ng[2318]: SSL error while reading stream; tls_error='SSL routines:SSL23_READ:ssl handshake failure' Jun 11 16:44:40 mmt-l-al36 syslog-ng[2318]: I/O error occurred while reading; fd='15', error='Connection reset by peer (104)' Jun 11 16:44:40 mmt-l-al36 syslog-ng[2318]: Syslog connection closed; fd='15', client='AF_INET(10.35.37.168:46993)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:42 mmt-l-al36 syslog-ng[2318]: Syslog connection accepted; fd='15', client='AF_INET(10.35.37.167:50371)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:42 mmt-l-al36 syslog-ng[2318]: SSL error while reading stream; tls_error='SSL routines:SSL23_READ:ssl handshake failure' Jun 11 16:44:42 mmt-l-al36 syslog-ng[2318]: I/O error occurred while reading; fd='15', error='Connection reset by peer (104)' Jun 11 16:44:42 mmt-l-al36 syslog-ng[2318]: Syslog connection closed; fd='15', client='AF_INET(10.35.37.167:50371)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:45 mmt-l-al36 syslog-ng[2318]: Syslog connection accepted; fd='15', client='AF_INET(10.35.37.168:47076)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:45 mmt-l-al36 syslog-ng[2318]: SSL error while reading stream; tls_error='SSL routines:SSL23_READ:ssl handshake failure' Jun 11 16:44:45 mmt-l-al36 syslog-ng[2318]: I/O error occurred while reading; fd='15', error='Connection reset by peer (104)' Jun 11 16:44:45 mmt-l-al36 syslog-ng[2318]: Syslog connection closed; fd='15', client='AF_INET(10.35.37.168:47076)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:47 mmt-l-al36 syslog-ng[2318]: Syslog connection accepted; fd='15', client='AF_INET(10.35.37.167:50457)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:47 mmt-l-al36 syslog-ng[2318]: SSL error while reading stream; tls_error='SSL routines:SSL23_READ:ssl handshake failure' Jun 11 16:44:47 mmt-l-al36 syslog-ng[2318]: I/O error occurred while reading; fd='15', error='Connection reset by peer (104)' Jun 11 16:44:47 mmt-l-al36 syslog-ng[2318]: Syslog connection closed; fd='15', client='AF_INET(10.35.37.167:50457)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:49 mmt-l-al36 syslog-ng[2318]: Log statistics; processed='destination(d_funambol)=13', processed='src.internal(s_balabit#0)=267517', stamp='src.internal(s_balabit#0)=1276267487', processed='source(s_balabit)=267517', processed='destination(d_gigamail)=4', processed='destination(d_webmail)=19', processed='destination(d_dovecot)=3408', processed='source(s_network)=9556565', processed='destination(d_balabit)=267517', processed='center(received)=0', processed='center(queued)=0', processed='destination(d_maillog)=14579' Jun 11 16:44:50 mmt-l-al36 syslog-ng[2318]: Syslog connection accepted; fd='15', client='AF_INET(10.35.37.168:47161)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:50 mmt-l-al36 syslog-ng[2318]: SSL error while reading stream; tls_error='SSL routines:SSL23_READ:ssl handshake failure' Jun 11 16:44:50 mmt-l-al36 syslog-ng[2318]: I/O error occurred while reading; fd='15', error='Connection reset by peer (104)' Jun 11 16:44:50 mmt-l-al36 syslog-ng[2318]: Syslog connection closed; fd='15', client='AF_INET(10.35.37.168:47161)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:52 mmt-l-al36 syslog-ng[2318]: Syslog connection accepted; fd='15', client='AF_INET(10.35.37.167:50540)', local='AF_INET(10.35.37.164:10001)' Jun 11 16:44:52 mmt-l-al36 syslog-ng[2318]: SSL error while reading stream; tls_error='SSL routines:SSL23_READ:ssl handshake failure' Jun 11 16:44:52 mmt-l-al36 syslog-ng[2318]: I/O error occurred while reading; fd='15', error='Connection reset by peer (104)' Jun 11 16:44:52 mmt-l-al36 syslog-ng[2318]: Syslog connection closed; fd='15', client='AF_INET(10.35.37.167:50540)', local='AF_INET(10.35.37.164:10001)' Client: Jun 10 12:49:52 mmt-l-fl13 syslog-ng[27982]: I/O error occurred while writing; fd='8', error='Connection reset by peer (104)' Jun 10 12:49:52 mmt-l-fl13 syslog-ng[27982]: Syslog connection broken; fd='8', server='AF_INET(10.35.37.166:10001)', time_reopen='10' Jun 10 12:50:02 mmt-l-fl13 syslog-ng[27982]: Syslog connection established; fd='8', server='AF_INET(10.35.37.166:10001)', local='AF_INET(0.0.0.0:0)' Jun 10 12:50:26 mmt-l-fl13 syslog-ng[27982]: Log statistics; processed='destination(d_network)=1', dropped='dst.tcp(d_network#0,syslog-aagg-vip:10001)=0', processed='dst.tcp(d_network#0,syslog-aagg-vip:10001)=1', stored='dst.tcp(d_network#0,syslog-aagg-vip:10001)=0', processed='source(s_syslog_pomezia)=2', processed='src.internal(s_balabit#0)=13', stamp='src.internal(s_balabit#0)=1276167002', processed='source(s_balabit)=13', dropped='dst.tcp(d_logcoll_ssl_pomezia#0,ssb-vip-pomezia.telecomitalia.it:514)=0', processed='dst.tcp(d_logcoll_ssl_pomezia#0,ssb-vip-pomezia.telecomitalia.it:514)=2', stored='dst.tcp(d_logcoll_ssl_pomezia#0,ssb-vip-pomezia.telecomitalia.it:514)=0', processed='source(s_funambol)=22', processed='destination(d_internal)=13', processed='destination(d_logcoll_ssl_pomezia)=2', processed='center(received)=0', processed='center(queued)=0' Jun 10 12:55:14 mmt-l-fl13 syslog-ng[27982]: Certificate subject matches configured hostname; hostname='syslog-aagg-vip', certificate='syslog-aagg-vip' Jun 10 13:00:26 mmt-l-fl13 syslog-ng[27982]: Log statistics; processed='destination(d_network)=3', dropped='dst.tcp(d_network#0,syslog-aagg-vip:10001)=0', processed='dst.tcp(d_network#0,syslog-aagg-vip:10001)=3', stored='dst.tcp(d_network#0,syslog-aagg-vip:10001)=0', processed='source(s_syslog_pomezia)=2', processed='src.internal(s_balabit#0)=15', stamp='src.internal(s_balabit#0)=1276167314', processed='source(s_balabit)=15', dropped='dst.tcp(d_logcoll_ssl_pomezia#0,ssb-vip-pomezia.telecomitalia.it:514)=0', processed='dst.tcp(d_logcoll_ssl_pomezia#0,ssb-vip-pomezia.telecomitalia.it:514)=2', stored='dst.tcp(d_logcoll_ssl_pomezia#0,ssb-vip-pomezia.telecomitalia.it:514)=0', processed='source(s_funambol)=65', processed='destination(d_internal)=15', processed='destination(d_logcoll_ssl_pomezia)=2', processed='center(received)=0', processed='center(queued)=0' Jun 10 13:07:43 mmt-l-fl13 syslog-ng[27982]: I/O error occurred while writing; fd='8', error='Connection reset by peer (104)' Jun 10 13:07:43 mmt-l-fl13 syslog-ng[27982]: Syslog connection broken; fd='8', server='AF_INET(10.35.37.166:10001)', time_reopen='10' Jun 10 13:07:53 mmt-l-fl13 syslog-ng[27982]: Syslog connection established; fd='8', server='AF_INET(10.35.37.166:10001)', local='AF_INET(0.0.0.0:0)' Jun 10 13:10:26 mmt-l-fl13 syslog-ng[27982]: Log statistics; processed='destination(d_network)=5', dropped='dst.tcp(d_network#0,syslog-aagg-vip:10001)=0', processed='dst.tcp(d_network#0,syslog-aagg-vip:10001)=5', stored='dst.tcp(d_network#0,syslog-aagg-vip:10001)=0', processed='source(s_syslog_pomezia)=2', processed='src.internal(s_balabit#0)=19', stamp='src.internal(s_balabit#0)=1276168073', processed='source(s_balabit)=19', dropped='dst.tcp(d_logcoll_ssl_pomezia#0,ssb-vip-pomezia.telecomitalia.it:514)=0', processed='dst.tcp(d_logcoll_ssl_pomezia#0,ssb-vip-pomezia.telecomitalia.it:514)=2', stored='dst.tcp(d_logcoll_ssl_pomezia#0,ssb-vip-pomezia.telecomitalia.it:514)=0', processed='source(s_funambol)=108', processed='destination(d_internal)=19', processed='destination(d_logcoll_ssl_pomezia)=2', processed='center(received)=0', processed='center(queued)=0' Thanks
participants (2)
-
Giovanni Mancuso
-
HÖLTZL Péter