Hello, After spending the afternoon trying to get this working, I've decided to reach out for some help (tried google - no luck!). I have what I believe is a very simple syslog-ng configuration where I am trying to capture syslog events from Cisco routers via TCP. I first tested via UDP, and all works good. The router establishes a TCP connection with my syslog-ng (viewable via netstat -nap, and in the /var/log/messages): Mar 25 17:07:53 logger syslog-ng[8943]: Syslog connection accepted; fd='11', client='AF_INET(X.X.X.254:56868)', local='AF_INET(X.X.X.5:2002)' On the router I do a "config term" & "exit" to generate a log. tcpdump on the syslog-ng machine shows traffic as expected: 17:18:44.891080 IP X.X.X.254.61241 > 10.240.0.5.2002: . 3679441367:3679441450(83) ack 1811354960 win 4128 17:18:44.891134 IP X.X.X.5.2002> 10.240.0.254.61241: . ack 83 win 5840 And a strace: poll([{fd=6, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=8, events=POLLIN}], 4, 456650) = 1 ([{fd=8, revents=POLLIN}]) gettimeofday({1269559238, 85688}, NULL) = 0 read(8, "<189>145: *Mar 25 23:41:37.033: "..., 7739) = 83 read(8, 0x9f94618, 7656) = -1 EAGAIN (Resource temporarily unavailable) gettimeofday({1269559238, 86032}, NULL) = 0 poll([{fd=6, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=8, events=POLLIN}], 4, 449113 But nothing ever shows up in the log files! Here's my syslog-ng.conf. Any pointers or ideas? @version: 3.0 options { use_dns(no); }; source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) ); }; source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; destination d_messages { file("/var/log/messages"); }; destination d_cisco { file("/var/log/cisco.log"); }; log { source(t_net); destination(d_messages); }; log { source(s_local); destination(d_messages); }; And for what it is worth the relevant lines from my test cisco: logging host X.X.X.5 transport tcp port 2002 And syslog-ng version: syslog-ng 3.0.5 Installer-Version: 3.0.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#f2fd7457627a88eb9602a73a5c16cd8b2434b667 Compile-Date: Dec 7 2009 14:52:42 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: off Thanks in advance!
Try adding the 'no-parse' flag to the source. Syslog-ng tries to parse out the headers of the message (like date/time, host, facility, etc), and if it cant figure out the format of the headers, it drops the message. The no-parse causes the entire message (headers and all if they exist) to get shoved into the message contents, and it generates new default headers. So source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) ); }; will become source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) flags('no-parse')); }; If the message does actually have headers, just syslog-ng cant understand them, you can use rewrite rules and 'set' statements to parse out the headers and set them manually. Sent: Thursday, March 25, 2010 5:31:15 PM From: d lists <dlists95@gmail.com> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] 3.0.5 & Cisco TCP problems
Hello,
After spending the afternoon trying to get this working, I've decided to reach out for some help (tried google - no luck!).
I have what I believe is a very simple syslog-ng configuration where I am trying to capture syslog events from Cisco routers via TCP. I first tested via UDP, and all works good. The router establishes a TCP connection with my syslog-ng (viewable via netstat -nap, and in the /var/log/messages):
Mar 25 17:07:53 logger syslog-ng[8943]: Syslog connection accepted; fd='11', client='AF_INET(X.X.X.254:56868)', local='AF_INET(X.X.X.5:2002)'
On the router I do a "config term" & "exit" to generate a log. tcpdump on the syslog-ng machine shows traffic as expected:
17:18:44.891080 IP X.X.X.254.61241 > 10.240.0.5.2002: . 3679441367:3679441450(83) ack 1811354960 win 4128 17:18:44.891134 IP X.X.X.5.2002> 10.240.0.254.61241: . ack 83 win 5840
And a strace:
poll([{fd=6, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=8, events=POLLIN}], 4, 456650) = 1 ([{fd=8, revents=POLLIN}]) gettimeofday({1269559238, 85688}, NULL) = 0 read(8, "<189>145: *Mar 25 23:41:37.033: "..., 7739) = 83 read(8, 0x9f94618, 7656) = -1 EAGAIN (Resource temporarily unavailable) gettimeofday({1269559238, 86032}, NULL) = 0 poll([{fd=6, events=POLLIN}, {fd=5, events=POLLIN}, {fd=3, events=POLLIN}, {fd=8, events=POLLIN}], 4, 449113
But nothing ever shows up in the log files! Here's my syslog-ng.conf. Any pointers or ideas?
@version: 3.0 options { use_dns(no); }; source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) ); }; source s_local { internal(); unix-stream("/dev/log"); file("/proc/kmsg" program_override("kernel: ")); }; destination d_messages { file("/var/log/messages"); }; destination d_cisco { file("/var/log/cisco.log"); }; log { source(t_net); destination(d_messages); }; log { source(s_local); destination(d_messages); };
And for what it is worth the relevant lines from my test cisco:
logging host X.X.X.5 transport tcp port 2002
And syslog-ng version:
syslog-ng 3.0.5 Installer-Version: 3.0.5 Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#f2fd7457627a88eb9602a73a5c16cd8b2434b667 Compile-Date: Dec 7 2009 14:52:42 Enable-Threads: on Enable-Debug: off Enable-GProf: off Enable-Memtrace: off Enable-Sun-STREAMS: off Enable-Sun-Door: off Enable-IPv6: on Enable-Spoof-Source: on Enable-TCP-Wrapper: off Enable-SSL: on Enable-SQL: on Enable-Linux-Caps: on Enable-Pcre: off
Thanks in advance! ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, Mar 25, 2010 at 6:59 PM, Patrick H. <syslogng@feystorm.net> wrote:
Try adding the 'no-parse' flag to the source. Syslog-ng tries to parse out the headers of the message (like date/time, host, facility, etc), and if it cant figure out the format of the headers, it drops the message. The no-parse causes the entire message (headers and all if they exist) to get shoved into the message contents, and it generates new default headers.
So source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) ); }; will become source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) flags('no-parse')); };
Tried that, no change. I've discovered what I think the problem is though: The cisco isn't including a LF at the end of each syslog message. If I force the router to send enough messages, a buffer must fill up & I get all the messages at once in a very unreadable format: Mar 25 20:28:20 10.240.0.254 <189>461: *Mar 26 02:45:22.244: %SYS-5-CONFIG_I: Configured from console by console<190>462: *Mar 26 02:45:28.244: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.240.0.5 port 2002 started - reconnection<189>463: *Mar 26 02:45:35.336: %SYS-5-CONFIG_I: Configured from console by console<189>464: *Mar 26 02:45:35.352: %SYS-5-CONFIG_I: Configured from console by console<189>465: *Mar 26 02:45:35.372: %SYS-5-CONFIG_I: Configured from console by console<189>...(repeat another 50 times at least) I found a thread for another piece of syslog software that encountered the same issue: http://www.gossamer-threads.com/lists/rsyslog/users/1204 I take it from the lack of people noticing this that there aren't too many people using TCP to gather syslog from Cisco routers. If anyone has some suggestions on possible solutions (outside of opening a TAC case with cisco - which I plan on doing), I am all ears. Thanks for the quick response! Time to read some more documentation.
If the message does actually have headers, just syslog-ng cant understand them, you can use rewrite rules and 'set' statements to parse out the headers and set them manually.
Sent: Thursday, March 25, 2010 5:31:15 PM From: d lists <dlists95@gmail.com> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] 3.0.5 & Cisco TCP problems
Hello,
After spending the afternoon trying to get this working, I've decided to reach out for some help (tried google - no luck!).
<snip>
Can the cisco use UDP? If so syslog-ng might consider each packet to be a complete message (since its connectionless) and flush it out. Try turning the keep-alive off as well. If the cisco is trying to close the connection after each message, that could also force it to flush. Just guesses at this point though. Sent: Thursday, March 25, 2010 8:31:54 PM From: d lists <dlists95@gmail.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] 3.0.5 & Cisco TCP problems
On Thu, Mar 25, 2010 at 6:59 PM, Patrick H. <syslogng@feystorm.net> wrote:
Try adding the 'no-parse' flag to the source. Syslog-ng tries to parse out the headers of the message (like date/time, host, facility, etc), and if it cant figure out the format of the headers, it drops the message. The no-parse causes the entire message (headers and all if they exist) to get shoved into the message contents, and it generates new default headers.
So source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) ); }; will become source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) flags('no-parse')); };
Tried that, no change. I've discovered what I think the problem is though: The cisco isn't including a LF at the end of each syslog message. If I force the router to send enough messages, a buffer must fill up & I get all the messages at once in a very unreadable format:
Mar 25 20:28:20 10.240.0.254 <189>461: *Mar 26 02:45:22.244: %SYS-5-CONFIG_I: Configured from console by console<190>462: *Mar 26 02:45:28.244: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.240.0.5 port 2002 started - reconnection<189>463: *Mar 26 02:45:35.336: %SYS-5-CONFIG_I: Configured from console by console<189>464: *Mar 26 02:45:35.352: %SYS-5-CONFIG_I: Configured from console by console<189>465: *Mar 26 02:45:35.372: %SYS-5-CONFIG_I: Configured from console by console<189>...(repeat another 50 times at least)
I found a thread for another piece of syslog software that encountered the same issue:
http://www.gossamer-threads.com/lists/rsyslog/users/1204
I take it from the lack of people noticing this that there aren't too many people using TCP to gather syslog from Cisco routers. If anyone has some suggestions on possible solutions (outside of opening a TAC case with cisco - which I plan on doing), I am all ears.
Thanks for the quick response! Time to read some more documentation.
If the message does actually have headers, just syslog-ng cant understand them, you can use rewrite rules and 'set' statements to parse out the headers and set them manually.
Sent: Thursday, March 25, 2010 5:31:15 PM From: d lists <dlists95@gmail.com> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] 3.0.5 & Cisco TCP problems
Hello,
After spending the afternoon trying to get this working, I've decided to reach out for some help (tried google - no luck!).
<snip> ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
On Thu, Mar 25, 2010 at 8:39 PM, Patrick H. <syslogng@feystorm.net> wrote:
Can the cisco use UDP? If so syslog-ng might consider each packet to be a complete message (since its connectionless) and flush it out. Try turning the keep-alive off as well. If the cisco is trying to close the connection after each message, that could also force it to flush. Just guesses at this point though.
Yes, UDP is the default (and works as expected). I'm trying to use TCP to gain a little more reliability with the delivery of the syslog.
Sent: Thursday, March 25, 2010 8:31:54 PM From: d lists <dlists95@gmail.com> To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] 3.0.5 & Cisco TCP problems
<snip>
Hi, d lists wrote:
On Thu, Mar 25, 2010 at 6:59 PM, Patrick H. <syslogng@feystorm.net> wrote:
Try adding the 'no-parse' flag to the source. Syslog-ng tries to parse out the headers of the message (like date/time, host, facility, etc), and if it cant figure out the format of the headers, it drops the message. The no-parse causes the entire message (headers and all if they exist) to get shoved into the message contents, and it generates new default headers.
So source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) ); }; will become source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) flags('no-parse')); };
Tried that, no change. I've discovered what I think the problem is though: The cisco isn't including a LF at the end of each syslog message. reducing the flush_timeout() may solve it (but it is just my idea). If I force the router to send enough messages, a buffer must fill up & I get all the messages at once in a very unreadable format:
Mar 25 20:28:20 10.240.0.254 <189>461: *Mar 26 02:45:22.244: %SYS-5-CONFIG_I: Configured from console by console<190>462: *Mar 26 02:45:28.244: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.240.0.5 port 2002 started - reconnection<189>463: *Mar 26 02:45:35.336: %SYS-5-CONFIG_I: Configured from console by console<189>464: *Mar 26 02:45:35.352: %SYS-5-CONFIG_I: Configured from console by console<189>465: *Mar 26 02:45:35.372: %SYS-5-CONFIG_I: Configured from console by console<189>...(repeat another 50 times at least)
At the moment, this kind of CISCO log's format is not supported by syslog-ng, because it's not syslog format (here is a sequence number instead of date). However, we are planning to solve it soon (perhaps in v3.2). Until then, I recommend that you should turn off using the sequence-number on CISCO router. You can do it with the following command: "no service sequence-numbers" (to turn on: "service sequence-numbers").
I found a thread for another piece of syslog software that encountered the same issue:
http://www.gossamer-threads.com/lists/rsyslog/users/1204
I take it from the lack of people noticing this that there aren't too many people using TCP to gather syslog from Cisco routers. If anyone has some suggestions on possible solutions (outside of opening a TAC case with cisco - which I plan on doing), I am all ears.
Thanks for the quick response! Time to read some more documentation.
If the message does actually have headers, just syslog-ng cant understand them, you can use rewrite rules and 'set' statements to parse out the headers and set them manually.
Sent: Thursday, March 25, 2010 5:31:15 PM From: d lists <dlists95@gmail.com> To: syslog-ng@lists.balabit.hu Subject: [syslog-ng] 3.0.5 & Cisco TCP problems
Hello,
After spending the afternoon trying to get this working, I've decided to reach out for some help (tried google - no luck!).
<snip> ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html
-- pzolee
On Thu, 2010-03-25 at 20:31 -0600, d lists wrote:
On Thu, Mar 25, 2010 at 6:59 PM, Patrick H. <syslogng@feystorm.net> wrote:
Try adding the 'no-parse' flag to the source. Syslog-ng tries to parse out the headers of the message (like date/time, host, facility, etc), and if it cant figure out the format of the headers, it drops the message. The no-parse causes the entire message (headers and all if they exist) to get shoved into the message contents, and it generates new default headers.
So source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) ); }; will become source t_net { tcp(ip(X.X.X.5) port(2002) keep-alive(yes) flags('no-parse')); };
Tried that, no change. I've discovered what I think the problem is though: The cisco isn't including a LF at the end of each syslog message. If I force the router to send enough messages, a buffer must fill up & I get all the messages at once in a very unreadable format:
Mar 25 20:28:20 10.240.0.254 <189>461: *Mar 26 02:45:22.244: %SYS-5-CONFIG_I: Configured from console by console<190>462: *Mar 26 02:45:28.244: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.240.0.5 port 2002 started - reconnection<189>463: *Mar 26 02:45:35.336: %SYS-5-CONFIG_I: Configured from console by console<189>464: *Mar 26 02:45:35.352: %SYS-5-CONFIG_I: Configured from console by console<189>465: *Mar 26 02:45:35.372: %SYS-5-CONFIG_I: Configured from console by console<189>...(repeat another 50 times at least)
That's right, the missing LF character causes syslog-ng not wait for more, and it only flushes the incoming message when it reaches the maximum specified by log_msg_size().
I found a thread for another piece of syslog software that encountered the same issue:
http://www.gossamer-threads.com/lists/rsyslog/users/1204
I take it from the lack of people noticing this that there aren't too many people using TCP to gather syslog from Cisco routers. If anyone has some suggestions on possible solutions (outside of opening a TAC case with cisco - which I plan on doing), I am all ears.
Thanks for the quick response! Time to read some more documentation.
The workaround outlined in that thread will cause problems, especially if you have application layer networking gear that processes the traffic before syslog-ng. (e.g. proxy based firewalls). TCP is stream based by definition, packet boundaries are not even ensured by the local TCP stack. The functionality is quite simple to add syslog-ng though, so I did that. There was a complication with our current config parser code, thus I don't intend to integrate it to our current stable version, but I may give a shot in 3.2. You can find the patch in cisco-tcp-workaround branch in the 3.0 git repository. More exactly: http://git.balabit.hu/?p=bazsi/syslog-ng-3.0.git;a=shortlog;h=cisco-tcp-work... Please report back if it did work for you, and make sure you use a separate listener for Cisco equipment that exhibits this behaviour. For other clients, this _will_ cause problems. -- Bazsi
On Thu, 2010-03-25 at 18:59 -0600, Patrick H. wrote:
Try adding the 'no-parse' flag to the source. Syslog-ng tries to parse out the headers of the message (like date/time, host, facility, etc), and if it cant figure out the format of the headers, it drops the message.
this is not true, it doesn't drop the message, it creates an 'Error processing message: <msg>' message, in order to see that there was indeed a message coming in. But judging the strace, the format of the message does include Cisco sequence number which is not supported by syslog-ng right now (but as PZolee has stated, we're working on that) -- Bazsi
participants (4)
-
Balazs Scheidler
-
d lists
-
Patrick H.
-
Zoltán Pallagi