syslog-ng truncating pipe template output
I've been trying to figure out an obscure problem with syslog-ng importing to a mysql database. I have various types of network devices feeding syslog-ng on local3 through local6. I can import from Cisco, UNIX servers, Windows - but not Netscreen firewalls! Each device type gets it's own mysql database. The mysql INSERT INTO statements for the Netscreen logs are truncated and hence fail to import because the mysql syntax is not correct. I chased wild geese for a while thinking the log format of the Netscreen was messing with mysql - but that's not the case. Notice it's just truncating the last few characters of each statement - including the all important ")" and "\n" newline that closes the mysql statement. I dumped an instance or two of these to a file instead of the normal fifo, added a ")" and a newline at the end of each, and it imported just fine! INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( 'X.X.X.X', 'local4', 'notice', 'notice', 'a5', '2004-11-19', '16:48:04', 'de-gw-1', 'de-gw-1: NetScreen device_id=de-gw-1 [No Name]system-notification-00257(traffic): start_time=\"2004-11-19 18:02:12\" duration=3 policy_id=2 service=NETBIOS (SSN) proto=6 src zone=Trust dst zone=Untrust action=Permit sent=1813 rcvd=5846 src=X.X.X.X dst=X.X.X.X src_port=3525 dst_port=139 src-xlated ip=X.X.X.X port=3525 INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( 'X.X.X.X', 'local4', 'notice', 'notice', 'a5', '2004-11-19', '16:48:06', 'de-gw-1', 'de-gw-1: NetScreen device_id=de-gw-1 [No Name]system-notification-00257(traffic): start_time=\"2004-11-19 18:02:14\" duration=3 policy_id=3 service=icmp proto=1 src zone=Untrust dst zone=Trust action=Permit sent=78 rcvd=78 src=X.X.X.X dst=X.X.X.X icmp type=8 INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( 'X.X.X.X', 'local4', 'notice', 'notice', 'a5', '2004-11-19', '16:48:06', 'de-gw-1', 'de-gw-1: NetScreen device_id=de-gw-1 [No Name]system-notification-00257(traffic): start_time=\"2004-11-19 18:01:15\" duration=62 policy_id=3 service=udp/port:3077 proto=17 src zone=Untrust dst zone=Trust action=Permit sent=0 rcvd=0 src=X.X.X.X dst=X.X.X.X src_port=17359 dst_port=3077 src-xlated ip=X.X.X.X port=17359 Remember that this is happening just for the Netscreen devices, using identical syslog-ng configs as for the other device types. Here's an example config: source net { udp(); }; filter f_firewalls { facility(local4); }; log { source(net); filter(f_firewalls); destination(firewalls); }; destination firewalls { pipe("/tmp/mysql_firewalls.pipe" template(" INSERT INTO logs (host, facility, priority, level, tag, date, time, program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL', '$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG ');\n") template-escape(yes)); }; This is on Gentoo Linux using Syslog-ng 1.6.4 stats(43200); log_msg_size(8192); log_fifo_size(8192); As per http://www.campin.net/syslog-ng/faq.html#message_length - is this the same issue? I could do some obscure perl filtering to fix this on the fly, but I'd rather fix the problem rather than the symptom. Thanks for your time. James Masson
I've been trying to figure out an obscure problem with syslog-ng importing to a mysql database.
This seems to be the same problem I posted about last week, with SpamAssassin. No answer yet.
As per http://www.campin.net/syslog-ng/faq.html#message_length - is this the same issue?
I don't see how it can be an input buffer length issue as each is way less than the 1k or 8k. Or a system library issue, as each log entry still has its INSERT statement (or timestamp in my case, with plaintext logs) which only syslog-ng adds, so it's obviously getting each line intact and separate. 2004-11-20T16:41:00+1000 chloe spamd[704]: connection from localhost [127.0.0.1] at port 33809 2004-11-20T16:41:01+1000 chloe spamd[23149]: info: setuid to slittle succeeded 2004-11-20T16:41:01+1000 chloe spamd[23149]: processing message <419ED8F2.3010106@slittle.com> for slittle:1000. 2004-11-20T16:41:01+1000 chloe spamd[23149]: clean message (-34.6/5.0) for slittle:1000 in 0.2 seconds, 562 bytes. 2004-11-20T16:41:01+1000 chloe qmail: 1100929261.295433 delivery 6355: success: did_2+0+1/ All one line.
On Fri, Nov 19, 2004 at 06:42:09PM +0100, James Masson wrote:
I've been trying to figure out an obscure problem with syslog-ng importing to a mysql database.
I have various types of network devices feeding syslog-ng on local3 through local6. I can import from Cisco, UNIX servers, Windows - but not Netscreen firewalls!
Each device type gets it's own mysql database. The mysql INSERT INTO statements for the Netscreen logs are truncated and hence fail to import because the mysql syntax is not correct.
Could you paste your entire config? Is your destination line in your config file a single line or broken up as in the email message, and if so is it broken up *exactly* as in the email message? I only ask because in the template I see a newline in the MSG macro like this: '$MSG ' -- Nate Without C, We would only have Pasal, Basi, and obol
On Fri, 2004-11-19 at 18:42, James Masson wrote:
I've been trying to figure out an obscure problem with syslog-ng importing to a mysql database.
I have various types of network devices feeding syslog-ng on local3 through local6. I can import from Cisco, UNIX servers, Windows - but not Netscreen firewalls!
Each device type gets it's own mysql database. The mysql INSERT INTO statements for the Netscreen logs are truncated and hence fail to import because the mysql syntax is not correct.
I chased wild geese for a while thinking the log format of the Netscreen was messing with mysql - but that's not the case. Notice it's just truncating the last few characters of each statement - including the all important ")" and "\n" newline that closes the mysql statement. I dumped an instance or two of these to a file instead of the normal fifo, added a ")" and a newline at the end of each, and it imported just fine!
IIRC there was a problem report about NetScreen logs including a NUL character somewhere in the middle of the message. That might cause this problem. Can you tcpdump an incoming UDP message as it reaches syslog-ng? I'd need the complete frame, so be sure to use the -s parameter for tcpdump. (specifying the maximum frame size, make sure it is at least the size of your MTU) -- Bazsi
participants (4)
-
Balazs Scheidler
-
James Masson
-
Nate Campi
-
scott