<html><head><meta http-equiv="Content-Type" content="text/html charset=iso-8859-1"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">In both test cases, I initiated the failure by restarting the syslog endpoint (which is actually a flume agent). When running through the ELB, the syslog-ng client never catches the connection failure and continues to try to send data through a TCP connection thats in CLOSE_WAIT state. When not using the ELB, the syslog-ng client notices immediately that the connection has failed and begins to reconnect in earnest.<div><br></div><div>--Matt</div><div><br><div><div>On May 7, 2013, at 9:29 PM, Balazs Scheidler <<a href="mailto:bazsi77@gmail.com">bazsi77@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><p dir="ltr">In both cases the client initiated the close operation not the load balancer nor the server. Where does the connection stall, then?</p>
<div class="gmail_quote">On May 7, 2013 11:17 PM, "Matt Wise" <<a href="mailto:matt@nextdoor.com">matt@nextdoor.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">Here's the dump THROUGH the ELB:<div><br></div><div><blockquote type="cite">21:11:26.208951 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [S], seq 267618391, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0</blockquote>
<blockquote type="cite">21:11:26.290452 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [S.], seq 848900027, ack 267618392, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 8], length 0</blockquote>
<blockquote type="cite">21:11:26.290509 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [.], ack 1, win 115, length 0</blockquote><blockquote type="cite">21:11:26.291460 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [P.], seq 1:227, ack 1, win 115, length 226</blockquote>
<blockquote type="cite">21:11:26.375765 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [.], ack 227, win 62, length 0</blockquote><blockquote type="cite">21:11:26.401850 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [.], seq 1:1461, ack 227, win 62, length 1460</blockquote>
<blockquote type="cite">21:11:26.401871 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [.], seq 1461:2921, ack 227, win 62, length 1460</blockquote><blockquote type="cite">
21:11:26.401898 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [P.], seq 2921:3515, ack 227, win 62, length 594</blockquote><blockquote type="cite">21:11:26.402343 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [.], ack 1461, win 137, length 0</blockquote>
<blockquote type="cite">21:11:26.402356 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [.], ack 2921, win 160, length 0</blockquote><blockquote type="cite">21:11:26.402361 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [.], ack 3515, win 183, length 0</blockquote>
<blockquote type="cite">21:11:26.484345 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [.], seq 227:3147, ack 3515, win 183, length 2920</blockquote><blockquote type="cite">
21:11:26.484365 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [P.], seq 3147:3690, ack 3515, win 183, length 543</blockquote><blockquote type="cite">21:11:26.566175 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [.], ack 3147, win 85, length 0 </blockquote>
<blockquote type="cite">21:11:26.569031 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [.], seq 3515:4975, ack 3690, win 96, length 1460</blockquote><blockquote type="cite">
21:11:26.569046 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [P.], seq 4975:5221, ack 3690, win 96, length 246</blockquote><blockquote type="cite">21:11:26.569222 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [.], ack 4975, win 206, length 0</blockquote>
<blockquote type="cite">21:11:26.569234 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [.], ack 5221, win 229, length 0</blockquote><blockquote type="cite">21:11:28.478081 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [P.], seq 3690:3727, ack 5221, win 229, length 37</blockquote>
<blockquote type="cite">21:11:28.603557 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [.], ack 3727, win 96, length 0 </blockquote><blockquote type="cite">21:11:50.707433 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [P.], seq 5221:5258, ack 3727, win 96, length 37</blockquote>
<blockquote type="cite">21:11:50.707460 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [.], ack 5258, win 229, length 0</blockquote><blockquote type="cite">21:11:50.707577 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [P.], seq 3727:3764, ack 5258, win 229, length 37</blockquote>
<blockquote type="cite">21:11:50.707599 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [F.], seq 3764, ack 5258, win 229, length 0</blockquote><blockquote type="cite">21:11:50.789084 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [.], ack 3764, win 96, length 0 </blockquote>
<blockquote type="cite">21:11:50.789847 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414: Flags [F.], seq 5258, ack 3765, win 96, length 0</blockquote><blockquote type="cite">21:11:50.789868 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.43414 > ELB.com.rfe: Flags [.], ack 5259, win 229, length 0</blockquote>
<div><br></div><div>Here's a direct connection:</div><div><br></div><div><blockquote type="cite">21:15:14.495542 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [S], seq 379756253, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0</blockquote>
<blockquote type="cite">21:15:14.576380 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [S.], seq 521570022, ack 379756254, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0</blockquote>
<blockquote type="cite">21:15:14.576409 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [.], ack 1, win 115, length 0</blockquote><blockquote type="cite">21:15:14.576940 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [P.], seq 1:227, ack 1, win 115, length 226</blockquote>
<blockquote type="cite">21:15:14.657397 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [.], ack 227, win 123, length 0</blockquote><blockquote type="cite">21:15:14.683465 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [.], seq 1:1461, ack 227, win 123, length 1460</blockquote>
<blockquote type="cite">21:15:14.683481 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [.], seq 1461:2921, ack 227, win 123, length 1460</blockquote><blockquote type="cite">
21:15:14.683485 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [P.], seq 2921:3515, ack 227, win 123, length 594</blockquote><blockquote type="cite">21:15:14.683683 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [.], ack 1461, win 137, length 0</blockquote>
<blockquote type="cite">21:15:14.683696 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [.], ack 2921, win 160, length 0</blockquote><blockquote type="cite">21:15:14.683702 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [.], ack 3515, win 183, length 0</blockquote>
<blockquote type="cite">21:15:14.766227 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [.], seq 227:3147, ack 3515, win 183, length 2920</blockquote><blockquote type="cite">
21:15:14.766243 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [P.], seq 3147:3690, ack 3515, win 183, length 543</blockquote><blockquote type="cite">21:15:14.846942 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [.], ack 3147, win 169, length 0</blockquote>
<blockquote type="cite">21:15:14.849068 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [.], seq 3515:4975, ack 3690, win 191, length 1460</blockquote><blockquote type="cite">
21:15:14.849082 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [P.], seq 4975:5221, ack 3690, win 191, length 246</blockquote><blockquote type="cite">21:15:14.849251 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [.], ack 4975, win 206, length 0</blockquote>
<blockquote type="cite">21:15:14.849262 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [.], ack 5221, win 229, length 0</blockquote><blockquote type="cite">21:15:18.394716 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [P.], seq 3690:3727, ack 5221, win 229, length 37</blockquote>
<blockquote type="cite">21:15:18.511442 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [.], ack 3727, win 191, length 0</blockquote><blockquote type="cite">21:15:52.957532 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [P.], seq 5221:5258, ack 3727, win 191, length 37</blockquote>
<blockquote type="cite">21:15:52.957587 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [.], ack 5258, win 229, length 0</blockquote><blockquote type="cite">21:15:52.957716 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [P.], seq 3727:3764, ack 5258, win 229, length 37</blockquote>
<blockquote type="cite">21:15:52.957742 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [F.], seq 3764, ack 5258, win 229, length 0</blockquote><blockquote type="cite">21:15:53.039203 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [.], ack 3764, win 191, length 0</blockquote>
<blockquote type="cite">21:15:53.039468 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [F.], seq 5258, ack 3764, win 191, length 0</blockquote><blockquote type="cite">21:15:53.039484 IP <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497 > ELB.com.rfe: Flags [.], ack 5259, win 229, length 0</blockquote>
<blockquote type="cite">21:15:53.039492 IP ELB.com.rfe > <a href="http://client.foo.com/" target="_blank">CLIENT.foo.com</a>.18497: Flags [.], ack 3765, win 191, length 0</blockquote></div><div><br></div><div>Any thoughts? By the way, I'm trying out 3.3.9, but running into other issues..</div>
<div><br></div><div><div>On May 7, 2013, at 1:55 PM, Balazs Scheidler <<a href="mailto:bazsi77@gmail.com" target="_blank">bazsi77@gmail.com</a>> wrote:</div><br><blockquote type="cite"><p dir="ltr"><br>
On May 7, 2013 10:51 PM, "Matt Wise" <<a href="mailto:matt@nextdoor.com" target="_blank">matt@nextdoor.com</a>> wrote:<br>
><br>
> I've done some more testing and now have narrowed the problem down to our Amazon ELB. Because the OSS version of Syslog-ng does not support failing over destinations from hostA to hostB when one fails, we are using an ELB in front of our syslog servers.<br>
><br>
> When we have no ELB in place, our syslog-ng client detects the network drop immediately and begins to try to reconnect. When the ELB is in the way, it never detects the network connection drop. I don't understand why. I've tested a bit manually using openssl to connect to our remote endpoint through the ELB and directly and I don't see any difference in the way network connections are killed off. Any thoughts here?<br>
><br></p><p dir="ltr">Hmm interesting. The difference might be how connections are terminated. Can you check that using tcpdump?</p><p dir="ltr">> --matt<br>
><br>
> On May 6, 2013, at 9:53 AM, Matt Wise <<a href="mailto:matt@nextdoor.com" target="_blank">matt@nextdoor.com</a>> wrote:<br>
><br>
> > We're running Syslog-NG 3.3.4 in our mixed Ubuntu 10/12 environment. We use SSL for all of our syslog-to-syslog connections, and have logging going to two different data pipelines.<br>
> ><br>
> > Data Dest #1: SyslogNG Client ----(SSL)----> SyslogNG Server ------> Logstash File-read-in-service<br>
> > Data Dest #2: SyslogNG Client ----(SSL)----> Stunnel Service ------> Flume Syslog Service<br>
> ><br>
> > The data streams work fine most of the time, but if we restart either the remote syslog-ng server, or the stunnel service, it seems that the syslog ng clients don't try to reconnect for a LONG time (or ever) to the endpoints again. I end up seeing the connection on the client go into a CLOSE_WAIT state, and syslog-ng keeps thinking that its sending log events through the connection, so it seems to never try to reconnect.<br>
> ><br>
> > I've tried setting time_reopen() to 0, 1 and 5... no luck or change in behavior.<br>
> ><br>
> > Any thoughts?<br>
> ><br>
> > --Matt<br>
> ><br>
><br>
> ______________________________________________________________________________<br>
> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
><br>
</p>
______________________________________________________________________________<br>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br></blockquote></div><br></div></div><br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq" target="_blank">http://www.balabit.com/wiki/syslog-ng-faq</a><br>
<br>
<br></blockquote></div>
______________________________________________________________________________<br>Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>FAQ: <a href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a><br><br></blockquote></div><br></div></body></html>