lol, thanks :-)<div>I'm actually a bit surprised that nobody has said anything before (like our customers).</div><div>Maybe they have, but I did a bug search and didn't find anything. I'll have to dig deeper.</div>
<div><br></div><div>I know the concept of tcp syslog is still relatively new, so maybe it just hasn't come up yet.</div><div>I'll still have to see if what they have was done on purpose/by design or if it is, in fact, a bug (or design flaw?). I dunno, hopefully someone smarter than me can answer that, which shouldn't be too hard, heh.</div>
<div><br></div><div><br clear="all">______________________________________________________________ <br><br>Clayton Dukes<br>______________________________________________________________<br>
<br><br><div class="gmail_quote">On Thu, Aug 19, 2010 at 3:26 PM, Matthew Hall <span dir="ltr"><<a href="mailto:mhall@mhcomputing.net">mhall@mhcomputing.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Thanks Clayton,<br>
<br>
I know how to explain things for networking people because I used to<br>
work at HP ProCurve. ;-)<br>
<br>
I really hope you can get this fixed because I think it will cause me<br>
problems at my new job.<br>
<font color="#888888"><br>
Matthew.<br>
</font><div><div></div><div class="h5"><br>
On Thu, Aug 19, 2010 at 03:13:24PM -0400, Clayton Dukes wrote:<br>
> Matt,<br>
> Thanks for that very good explanation, it's good ammo for me to approach the<br>
> developers @ Cisco with :-)<br>
><br>
> ______________________________________________________________<br>
><br>
> Clayton Dukes<br>
> ______________________________________________________________<br>
><br>
><br>
> On Thu, Aug 19, 2010 at 1:21 PM, Matthew Hall <<a href="mailto:mhall@mhcomputing.net">mhall@mhcomputing.net</a>> wrote:<br>
><br>
> > This understanding is Not quite right. What we are talking about here is<br>
> > the difference between SOCK_STREAM and SOCK_DGRAM socket semantics.<br>
> ><br>
> > In a SOCK_DGRAM each PDU or segment is considered to be a separate<br>
> > atomic message, and each is delivered to the application separately. For<br>
> > a SOCK_DGRAM, a connection just means you have bound the socket to a<br>
> > specific remote address. It does not open and close for each message<br>
> > because it is not connection oriented.<br>
> ><br>
> > In a SOCK_STREAM each PDU or segment is considered to be additional<br>
> > bytes in a continuous flow of data. The application which receives data<br>
> > from a SOCK_STREAM has no way of seeing each PDU or segment<br>
> > individually, because the PDUs or segments are reassembled inside the<br>
> > kernel.<br>
> ><br>
> > There is no standard way of simply receiving each SOCK_STREAM packet<br>
> > separately because there is no relationship between the size of each<br>
> > PDU or segment and the size of the message. The size of the message is<br>
> > defined by the '\n'.<br>
> ><br>
> > This difference is also seen in UDP DNS versus TCP DNS. TCP DNS sends<br>
> > extra fields in the packets to keep track of the message sizes and UDP<br>
> > DNS does not.<br>
> ><br>
> > Matthew.<br>
> ><br>
> > On Thu, Aug 19, 2010 at 11:34:39AM -0400, Clayton Dukes wrote:<br>
> > > bing! (light bulb just went off).<br>
> > > I wasn't thinking about the whole established vs. non established<br>
> > connection<br>
> > > thing.<br>
> > > I couldn't figure out why, if UDP wasn't sending a newline, it wasn't<br>
> > > causing the same problem. I wasn't thinking about the fact that the<br>
> > > connection was closing, thus ending the stream.<br>
> > ><br>
> > > Thanks for the help!<br>
> > ><br>
> > > ______________________________________________________________<br>
> > ><br>
> > > Clayton Dukes<br>
> > > ______________________________________________________________<br>
> > ><br>
> > ><br>
> > > On Thu, Aug 19, 2010 at 11:29 AM, <<a href="mailto:syslogng@feystorm.net">syslogng@feystorm.net</a>> wrote:<br>
> > ><br>
> > > > I explained it already :-)<br>
> > > > When the message comes in over TCP and doesnt end with a newline,<br>
> > syslog-ng<br>
> > > > assumes the message is going to be continued in another packet. When<br>
> > the<br>
> > > > cumulative total of all the messages exceeds the max message size it<br>
> > flushes<br>
> > > > the buffer out and you get all the messages mashed together at once.<br>
> > > > You can try filing a bug report on <a href="http://bugzilla.balabit.com" target="_blank">bugzilla.balabit.com</a> and request a<br>
> > new<br>
> > > > flag or something that treats each packet on a tcp source as a separate<br>
> > > > message, but I'd say the problem is more cisco than syslog-ng since<br>
> > > > syslog-ng works fine with all other sources except cisco devices :-/<br>
> > > > Look at it this way, every thing that sends logs out to tcp expects the<br>
> > > > receiving syslog daemon to treat a packet without a newline as a<br>
> > message to<br>
> > > > be continued in a later packet. If syslog-ng changed that default<br>
> > behavior,<br>
> > > > all these other things that expect the behavior would break.<br>
> > > ><br>
> > > > -Patrick<br>
> > > ><br>
> > > ><br>
> > > > Sent: Thursday, August 19, 2010 9:12:36 AM<br>
> > > ><br>
> > > > From: Clayton Dukes <<a href="mailto:cdukes@gmail.com">cdukes@gmail.com</a>> <<a href="mailto:cdukes@gmail.com">cdukes@gmail.com</a>><br>
> > > > To: Syslog-ng users' and developers' mailing list<br>
> > > > <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>> <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
> > > > Subject: Re: [syslog-ng] TCP recv bug in syslog-ng v2.09?<br>
> > > ><br>
> > > > Following up on this:<br>
> > > ><br>
> > > > Interestingly, however, is that when I use tcpdump, I DO see each<br>
> > message<br>
> > > > coming in via TCP...so now I'm leaning back towards syslog-ng being the<br>
> > > > problem.<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > Here's what I can see:<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > UDP:<br>
> > > ><br>
> > > > #tcpdump -vvv host 14.3.23.50 -i eth0<br>
> > > ><br>
> > > > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size<br>
> > 96<br>
> > > > bytes<br>
> > > ><br>
> > > > 09:13:31.465239 IP (tos 0xb8, ttl 251, id 1323, offset 0, flags [none],<br>
> > > > proto UDP (17), length 134)<br>
> > > ><br>
> > > > 14.3.23.50.51526 > server.x.com.syslog: SYSLOG, length: 106<br>
> > > ><br>
> > > > Facility local7 (23), Severity notice (5)<br>
> > > ><br>
> > > > Msg: 2975: *Aug 19 12:34:19.465: %SYS-5-CONFIG_I[|syslog]<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > TCP:<br>
> > > ><br>
> > > > #tcpdump -vvv host 14.3.23.50 -i eth0<br>
> > > ><br>
> > > > 09:46:29.902063 IP (tos 0x0, ttl 251, id 12253, offset 0, flags [none],<br>
> > > > proto TCP (6), length 146)<br>
> > > ><br>
> > > > 14.3.23.50.31746 > server.x.com.601: Flags [.], seq 233:339, ack 1,<br>
> > win<br>
> > > > 4128, length 106<br>
> > > ><br>
> > > > 09:46:29.902077 IP (tos 0x0, ttl 64, id 27779, offset 0, flags [DF],<br>
> > proto<br>
> > > > TCP (6), length 40)<br>
> > > ><br>
> > > > server.x.com.601 > 14.3.23.50.31746: Flags [.], cksum 0xa3bb<br>
> > (correct),<br>
> > > > seq 1, ack 339, win 5840, length 0<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > So, it looks like the syslog message does end, but why is syslog-ng<br>
> > > > buffering and showing multiple TCP-based messages as a single message?<br>
> > > ><br>
> > > > Do I have something misconfigured?<br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > > ______________________________________________________________<br>
> > > ><br>
> > > > Clayton Dukes<br>
> > > > ______________________________________________________________<br>
> > > ><br>
> > > ><br>
> > > > On Tue, Aug 17, 2010 at 3:14 PM, Clayton Dukes <<a href="mailto:cdukes@gmail.com">cdukes@gmail.com</a>><br>
> > wrote:<br>
> > > ><br>
> > > >> yikes!<br>
> > > >> seriously?<br>
> > > >> Guess I'll have to file a bug internally :-)<br>
> > > >> Can someone else positively verify this?<br>
> > > >> Or any suggestions on how I can so that we can recreate it in a lab?<br>
> > > >><br>
> > > >> ______________________________________________________________<br>
> > > >><br>
> > > >> Clayton Dukes<br>
> > > >> ______________________________________________________________<br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >> On Tue, Aug 17, 2010 at 2:52 PM, <<a href="mailto:syslogng@feystorm.net">syslogng@feystorm.net</a>> wrote:<br>
> > > >><br>
> > > >>> If I recall correctly its because cisco equipment doesnt terminate<br>
> > its<br>
> > > >>> log entries with newlines, so when sending via TCP, syslog-ng thinks<br>
> > the<br>
> > > >>> message is going to be continued in another packet (UDP is assumed to<br>
> > be 1<br>
> > > >>> packet per log entry).<br>
> > > >>> The only way to fix this is an ugly hack to set the timeout so that<br>
> > when<br>
> > > >>> it doesnt get a reply within a certain time, it assumes the log entry<br>
> > ended.<br>
> > > >>> but if several log entries are sent within the timeout, then they'll<br>
> > all be<br>
> > > >>> mashed together into 1 syslog-ng entry.<br>
> > > >>><br>
> > > >>><br>
> > > >>> Sent: Tuesday, August 17, 2010 12:28:28 PM<br>
> > > >>> From: Clayton Dukes <<a href="mailto:cdukes@gmail.com">cdukes@gmail.com</a>> <<a href="mailto:cdukes@gmail.com">cdukes@gmail.com</a>><br>
> > > >>> To: Syslog-ng users' and developers' mailing list<br>
> > > >>> <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>> <<a href="mailto:syslog-ng@lists.balabit.hu">syslog-ng@lists.balabit.hu</a>><br>
> > > >>> Subject: [syslog-ng] TCP recv bug in syslog-ng v2.09?<br>
> > > >>><br>
> > > >>> Hey guys,<br>
> > > >>> Are there any known bugs for syslog-ng v2.09 that won't allow a<br>
> > cisco<br>
> > > >>> router to send logs over tcp?<br>
> > > >>> I can see a connection established in syslog-ng.<br>
> > > >>> I also see the message come in via tcpdump, but nothing in<br>
> > syslog-ng's<br>
> > > >>> output.<br>
> > > >>> If I change the router from tcp to udp, messages come in as expected.<br>
> > > >>><br>
> > > >>> *Router config:*<br>
> > > >>><br>
> > > >>> logging source-interface Loopback0<br>
> > > >>> logging 172.18.224.150<br>
> > > >>> logging host 172.18.224.190 transport tcp<br>
> > > >>><br>
> > > >>><br>
> > > >>> *syslog-ng config:*<br>
> > > >>><br>
> > > >>> source s_all {<br>
> > > >>> udp();<br>
> > > >>> tcp(ip(11.31.130.99) port(8002) max-connections(300));<br>
> > > >>> tcp(ip(172.18.224.190) port(601) max-connections(300));<br>
> > > >>> };<br>
> > > >>><br>
> > > >>><br>
> > > >>> *debug output:*<br>
> > > >>> I commented out the line above for the other interface<br>
> > (11.31.130.99),<br>
> > > >>> restarted and this is all I see:<br>
> > > >>> Syslog connection accepted; from='AF_INET(14.3.23.50:63845)',<br>
> > > >>> to='AF_INET(<a href="http://172.18.224.190:601" target="_blank">172.18.224.190:601</a>)'<br>
> > > >>><br>
> > > >>><br>
> > > >>> *tcpdump:*<br>
> > > >>><br>
> > > >>> 14:13:46.914566 IP (tos 0x0, ttl 251, id 4303, offset 0, flags<br>
> > [none],<br>
> > > >>> proto TCP (6), length 134)<br>
> > > >>> 14.3.23.50.63845 > xxx.com.601: Flags [.], seq 230:324, ack 1,<br>
> > win<br>
> > > >>> 4128, length 94<br>
> > > >>><br>
> > > >>><br>
> > > >>> *Router debug:*<br>
> > > >>><br>
> > > >>> *Aug 17 17:34:19.772: %SYS-5-CONFIG_I: Configured from console by<br>
> > pnoc<br>
> > > >>> on vty0 (172.18.224.151)<br>
> > > >>> *Aug 17 17:34:20.776: Released port 15205 in Transport Port Agent for<br>
> > > >>> TCP IP type 1 delay 240000<br>
> > > >>> *Aug 17 17:34:20.776: TCB 0x850F9754 destroyed<br>
> > > >>> *Aug 17 17:34:25.775: TCB83648E60 created<br>
> > > >>> *Aug 17 17:34:25.775: TCB83648E60 setting property TCP_PID (8)<br>
> > 845083E4<br>
> > > >>> *Aug 17 17:34:25.775: TCB83648E60 setting property TCP_NO_DELAY (1)<br>
> > > >>> 845083E8<br>
> > > >>> *Aug 17 17:34:25.775: TCB83648E60 setting property TCP keepalive<br>
> > timeout<br>
> > > >>> (17) 845084A0<br>
> > > >>> *Aug 17 17:34:25.775: TCP: Random local port generated 63845, network<br>
> > 1<br>
> > > >>> *Aug 17 17:34:25.775: TCB83648E60 bound to 14.3.23.50.63845<br>
> > > >>> *Aug 17 17:34:25.775: Reserved port 63845 in Transport Port Agent for<br>
> > > >>> TCP IP type 1<br>
> > > >>> *Aug 17 17:34:25.775: TCP: sending SYN, seq 3300233565, ack 0<br>
> > > >>> *Aug 17 17:34:25.775: TCP0: Connection to <a href="http://172.18.224.190:601" target="_blank">172.18.224.190:601</a>,<br>
> > > >>> advertising MSS 536<br>
> > > >>> *Aug 17 17:34:25.775: TCP0: state was CLOSED -> SYNSENT [63845 -><br>
> > > >>> 172.18.224.190(601)]<br>
> > > >>> *Aug 17 17:34:25.779: TCP0: state was SYNSENT -> ESTAB [63845 -><br>
> > > >>> 172.18.224.190(601)]<br>
> > > >>> *Aug 17 17:34:25.779: TCP: tcb 83648E60 connection to<br>
> > <a href="http://172.18.224.190:601" target="_blank">172.18.224.190:601</a>,<br>
> > > >>> peer MSS 1460, MSS is 536<br>
> > > >>> *Aug 17 17:34:25.779: TCB83648E60 connected to 172.18.224.190.601<br>
> > > >>> *Aug 17 17:34:25.779: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host<br>
> > > >>> 172.18.224.190 port 601 started - reconnection<br>
> > > >>><br>
> > > >>> ______________________________________________________________<br>
> > > >>><br>
> > > >>> Clayton Dukes<br>
> > > >>> ______________________________________________________________<br>
> > > >>><br>
> > > >>> ------------------------------<br>
> > > >>><br>
> > > >>><br>
> > ______________________________________________________________________________<br>
> > > >>> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> > > >>> Documentation:<br>
> > <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> > > >>> FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
> > > >>><br>
> > > >>><br>
> > > >>><br>
> > > >>><br>
> > ______________________________________________________________________________<br>
> > > >>> Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> > > >>> Documentation:<br>
> > > >>> <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> > > >>> FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
> > > >>><br>
> > > >>><br>
> > > >>><br>
> > > >><br>
> > > > ------------------------------<br>
> > > ><br>
> > > ><br>
> > ______________________________________________________________________________<br>
> > > > Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> > > > Documentation:<br>
> > <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> > > > FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > ______________________________________________________________________________<br>
> > > > Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> > > > Documentation:<br>
> > > > <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> > > > FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
> > > ><br>
> > > ><br>
> > > ><br>
> ><br>
> > ><br>
> > ______________________________________________________________________________<br>
> > > Member info: <a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
> > > Documentation:<br>
> > <a href="http://www.balabit.com/support/documentation/?product=syslog-ng" target="_blank">http://www.balabit.com/support/documentation/?product=syslog-ng</a><br>
> > > FAQ: <a href="http://www.campin.net/syslog-ng/faq.html" target="_blank">http://www.campin.net/syslog-ng/faq.html</a><br>
> > ><br>
> ><br>
> ><br>
</div></div></blockquote></div><br></div>