bing! (light bulb just went off).<div>I wasn't thinking about the whole established vs. non established connection thing.</div><div>I couldn't figure out why, if UDP wasn't sending a newline, it wasn't causing the same problem. I wasn't thinking about the fact that the connection was closing, thus ending the stream.</div>
<div><br></div><div>Thanks for the help!</div><div><br clear="all">______________________________________________________________ <br><br>Clayton Dukes<br>______________________________________________________________<br>
<br><br><div class="gmail_quote">On Thu, Aug 19, 2010 at 11:29 AM, <span dir="ltr"><<a href="mailto:syslogng@feystorm.net">syslogng@feystorm.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<div bgcolor="#ffffff" text="#0050d0">
<font size="-1"><font face="Helvetica, Arial, sans-serif">I explained
it already :-)<br>
When the message comes in over TCP and doesnt end with a newline,
syslog-ng assumes the message is going to be continued in another
packet. When the cumulative total of all the messages exceeds the max
message size it flushes the buffer out and you get all the messages
mashed together at once.<br>
You can try filing a bug report on <a href="http://bugzilla.balabit.com" target="_blank">bugzilla.balabit.com</a> and request a
new flag or something that treats each packet on a tcp source as a
separate message, but I'd say the problem is more cisco than syslog-ng
since syslog-ng works fine with all other sources except cisco devices
:-/<br>
Look at it this way, every thing that sends logs out to tcp expects the
receiving syslog daemon to treat a packet without a newline as a
message to be continued in a later packet. If syslog-ng changed that
default behavior, all these other things that expect the behavior would
break.<br>
<br>
-Patrick<br>
</font></font><br>
<br>
Sent: Thursday, August 19, 2010 9:12:36 AM<div class="im"><br>
From: Clayton Dukes <a href="mailto:cdukes@gmail.com" target="_blank"><cdukes@gmail.com></a><br>
To: Syslog-ng users' and developers' mailing list
<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank"><syslog-ng@lists.balabit.hu></a> <br></div>
Subject: Re: [syslog-ng] TCP recv bug in syslog-ng v2.09?
<div><div></div><div class="h5"><blockquote type="cite">Following up on this:
<div><br>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">Interestingly,
however, is that when I use tcpdump, I DO see
each message coming in via TCP...so now I'm leaning back towards
syslog-ng
being the problem.</span></p>
<p class="MsoNormal"><span style="font-family:Calibri,sans-serif;font-size:15px;color:rgb(31, 73, 125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">Here's what I can
see:</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">UDP:</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">#tcpdump -vvv host
14.3.23.50 -i
eth0
</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">tcpdump: listening
on eth0, link-type EN10MB (Ethernet), capture
size 96 bytes</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">09:13:31.465239 IP
(tos 0xb8, ttl 251, id 1323, offset 0, flags
[none], proto UDP (17), length 134)</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> 14.3.23.50.51526
> server.x.com.syslog:
SYSLOG, length: 106</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> Facility
local7 (23),
Severity notice (5)</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> Msg: 2975:
*Aug 19
12:34:19.465: %SYS-5-CONFIG_I[|syslog]</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">TCP:</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">#tcpdump -vvv host
14.3.23.50 -i
eth0
</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">09:46:29.902063 IP
(tos 0x0, ttl 251, id 12253, offset 0, flags
[none], proto TCP (6), length 146)</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> 14.3.23.50.31746
>
server.x.com.601: Flags [.], seq 233:339, ack 1, win 4128, length 106</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">09:46:29.902077 IP
(tos 0x0, ttl 64, id 27779, offset 0, flags
[DF], proto TCP (6), length 40)</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> server.x.com.601
>
14.3.23.50.31746: Flags [.], cksum 0xa3bb (correct), seq 1, ack 339,
win 5840,
length 0</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"> </span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">So, it looks like
the syslog message does end, but why is
syslog-ng buffering and showing multiple TCP-based messages as a single
message?</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)">Do I have something
misconfigured?</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"><br>
</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"><br>
</span></p>
<p class="MsoNormal"><span style="font-size:11pt;color:rgb(31, 73, 125)"><br>
</span></p>
______________________________________________________________ <br>
<br>
Clayton Dukes<br>
______________________________________________________________<br>
<br>
<br>
<div class="gmail_quote">On Tue, Aug 17, 2010 at 3:14 PM, Clayton
Dukes <span dir="ltr"><<a href="mailto:cdukes@gmail.com" target="_blank">cdukes@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex">yikes!
<div>seriously?</div>
<div>Guess I'll have to file a bug internally :-)</div>
<div>Can someone else positively verify this?</div>
<div>Or any suggestions on how I can so that we can recreate it in
a lab?</div>
<div><br clear="all">
<font color="#888888">______________________________________________________________
<br>
<br>
Clayton Dukes<br>
______________________________________________________________</font>
<div>
<div><br>
<br>
<br>
<div class="gmail_quote">On Tue, Aug 17, 2010 at 2:52 PM, <span dir="ltr"><<a href="mailto:syslogng@feystorm.net" target="_blank">syslogng@feystorm.net</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="border-left:1px solid rgb(204, 204, 204);margin:0pt 0pt 0pt 0.8ex;padding-left:1ex">
<div bgcolor="#ffffff" text="#0050d0"><font size="-1"><font face="Helvetica, Arial, sans-serif">If I recall
correctly its because cisco equipment doesnt terminate its log entries
with newlines, so when sending via TCP, syslog-ng thinks the message is
going to be continued in another packet (UDP is assumed to be 1 packet
per log entry).<br>
The only way to fix this is an ugly hack to set the timeout so that
when it doesnt get a reply within a certain time, it assumes the log
entry ended. but if several log entries are sent within the timeout,
then they'll all be mashed together into 1 syslog-ng entry.<br>
<br>
</font></font><br>
Sent: Tuesday, August 17, 2010 12:28:28 PM<br>
From: Clayton Dukes <a href="mailto:cdukes@gmail.com" target="_blank"><cdukes@gmail.com></a><br>
To: Syslog-ng users' and developers' mailing list
<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank"><syslog-ng@lists.balabit.hu></a> <br>
Subject: [syslog-ng] TCP recv bug in syslog-ng v2.09?
<blockquote type="cite">
<div>
<div>Hey guys,
<div>Are there any known bugs for syslog-ng v2.09 that won't
allow a
cisco router to send logs over tcp?</div>
<div>I can see a connection established in syslog-ng.</div>
<div>I also see the message come in via tcpdump, but nothing in
syslog-ng's output.</div>
<div>If I change the router from tcp to udp, messages come in
as
expected.</div>
<div><br>
</div>
<div><b>Router config:</b></div>
<div><br>
</div>
<div>logging source-interface Loopback0 </div>
<div>logging <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">172.18.224.150</a> </div>
<div>logging host <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">172.18.224.190</a> transport tcp</div>
<div><br>
</div>
<div><br>
</div>
<div><b>syslog-ng config:</b></div>
<div><br>
</div>
<div>
<div>source s_all {</div>
<div> udp();</div>
<div> tcp(ip(11.31.130.99) port(8002)
max-connections(300));</div>
<div> tcp(ip(172.18.224.190) port(601)
max-connections(300));</div>
<div>};</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><b>debug output:</b></div>
<div>I commented out the line above for the other interface
(11.31.130.99), restarted and this is all I see:</div>
<div>Syslog connection accepted; from='AF_INET<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">(14.3.23.50</a>:63845)', to='AF_INET<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">(172.18.224.190</a>:601)'<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><b>tcpdump:</b></div>
<div><br>
</div>
<div>
<div>14:13:46.914566 IP (tos 0x0, ttl 251, id 4303, offset 0,
flags
[none], proto TCP (6), length 134)</div>
<div> 14.3.23.50.63845 > xxx.com.601: Flags [.], seq
230:324,
ack 1, win 4128, length 94</div>
<div><br>
</div>
<div><br>
</div>
<div><b>Router debug:</b></div>
<div><br>
</div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:19<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.772</a>:
%SYS<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">-5-</a>CONFIG_I:
Configured from console by pnoc on vty<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">0 (172.18.224.151)</a> </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:20<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.776</a>:
Released port <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">15205</a>
in Transport Port Agent for TCP IP type 1 delay <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">240000</a> </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:20<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.776</a>:
TCB 0x<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">850</a>F<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">9754</a>
destroyed </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
TCB<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">83648</a>E60
created </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
TCB<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">83648</a>E60
setting property TCP_PID <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">(8) 845083</a>E4</div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
TCB<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">83648</a>E60
setting property TCP_NO_DELAY <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">(1) 845083</a>E8</div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
TCB<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">83648</a>E60
setting property TCP keepalive timeout <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">(17) 845084</a>A0 </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
TCP: Random local port generated <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">63845</a>,
network 1 </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
TCB<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">83648</a>E60
bound to <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">14.3.23.50.63845</a> </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
Reserved port <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">63845</a>
in Transport Port Agent for TCP IP type 1 </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
TCP: sending SYN, seq <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">3300233565</a>, ack 0 </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
TCP0: Connection to <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">172.18.224.190</a>:601, advertising MSS <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">536</a> </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.775</a>:
TCP0: state was CLOSED -> SYNSENT [<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">63845 -</a>> <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">172.18.224.190(601)</a>] </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.779</a>:
TCP0: state was SYNSENT -> ESTAB [<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">63845 -</a>> <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">172.18.224.190(601)</a>] </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.779</a>:
TCP: tcb <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">83648</a>E60
connection to <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">172.18.224.190</a>:601, peer MSS <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">1460</a>,
MSS is <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">536</a> </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.779</a>:
TCB<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">83648</a>E60
connected to <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">172.18.224.190.601</a> </div>
<div>*Aug <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">17
17</a>:34:25<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">.779</a>:
%SYS<a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">-6-</a>LOGGINGHOST_STARTSTOP:
Logging to host <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">172.18.224.190</a> port <a style="border-bottom:1px solid;color:rgb(0, 102, 204);text-decoration:none" title="">601</a>
started - reconnection<br>
</div>
<div><br>
</div>
______________________________________________________________ <br>
<br>
Clayton Dukes<br>
______________________________________________________________<br>
</div>
</div>
</div>
<pre><hr size="4" width="90%">
______________________________________________________________________________
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
</div>
<br>
______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<pre><hr size="4" width="90%">
______________________________________________________________________________
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a>
</pre>
</blockquote>
</div></div></div>
<br>______________________________________________________________________________<br>
Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
Documentation: <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
<br>
<br></blockquote></div><br></div>