[syslog-ng] 3.0.5 & Cisco TCP problems

[Balazs Scheidler]

On Thu, 2010-03-25 at 20:31 -0600, d lists wrote:
> On Thu, Mar 25, 2010 at 6:59 PM, Patrick H. <syslogng at feystorm.net> wrote:
> > Try adding the 'no-parse' flag to the source. Syslog-ng tries to parse out
> > the headers of the message (like date/time, host, facility, etc), and if it
> > cant figure out the format of the headers, it drops the message. The
> > no-parse causes the entire message (headers and all if they exist) to get
> > shoved into the message contents, and it generates new default headers.
> >
> > So
> > source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) ); };
> > will become
> > source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes)
> > flags('no-parse')); };
> 
> Tried that, no change.  I've discovered what I think the problem is
> though:  The cisco isn't including a LF at the end of each syslog
> message.  If I force the router to send enough messages, a buffer must
> fill up & I get all the messages at once in a very unreadable format:
> 
> Mar 25 20:28:20 10.240.0.254 <189>461: *Mar 26 02:45:22.244:
> %SYS-5-CONFIG_I: Configured from console by console<190>462: *Mar 26
> 02:45:28.244: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.240.0.5
> port 2002 started - reconnection<189>463: *Mar 26 02:45:35.336:
> %SYS-5-CONFIG_I: Configured from console by console<189>464: *Mar 26
> 02:45:35.352: %SYS-5-CONFIG_I: Configured from console by
> console<189>465: *Mar 26 02:45:35.372: %SYS-5-CONFIG_I: Configured
> from console by console<189>...(repeat another 50 times at least)

That's right, the missing LF character causes syslog-ng not wait for
more, and it only flushes the incoming message when it reaches the
maximum specified by log_msg_size().

> 
> I found a thread for another piece of syslog software that encountered
> the same issue:
> 
> http://www.gossamer-threads.com/lists/rsyslog/users/1204
> 
> I take it from the lack of people noticing this that there aren't too
> many people using TCP to gather syslog from Cisco routers.  If anyone
> has some suggestions on possible solutions (outside of opening a TAC
> case with cisco - which I plan on doing), I am all ears.
> 
> Thanks for the quick response!  Time to read some more documentation.

The workaround outlined in that thread will cause problems, especially
if you have application layer networking gear that processes the traffic
before syslog-ng. (e.g. proxy based firewalls).

TCP is stream based by definition, packet boundaries are not even
ensured by the local TCP stack.

The functionality is quite simple to add syslog-ng though, so I did
that. There was a complication with our current config parser code, thus
I don't intend to integrate it to our current stable version, but I may
give a shot in 3.2.

You can find the patch in cisco-tcp-workaround branch in the 3.0 git
repository.

More exactly:

http://git.balabit.hu/?p=bazsi/syslog-ng-3.0.git;a=shortlog;h=cisco-tcp-workaround

Please report back if it did work for you, and make sure you use a
separate listener for Cisco equipment that exhibits this behaviour. For
other clients, this _will_ cause problems.



-- 
Bazsi