[syslog-ng]syslog-ng truncating pipe template output

[James Masson]

I've been trying to figure out an obscure problem with syslog-ng
importing to a mysql database.

I have various types of network devices feeding syslog-ng on local3
through local6. I can import from Cisco, UNIX servers, Windows - but not
Netscreen firewalls!

Each device type gets it's own mysql database. The mysql INSERT INTO
statements for the Netscreen logs are truncated and hence fail to import
because the mysql syntax is not correct.

 I chased wild geese for a while thinking the log format of the
Netscreen was messing with mysql - but that's not the case. Notice it's
just truncating the last few characters of each statement - including
the all important ")" and "\n" newline that closes the mysql statement.
I dumped an instance or two of these to a file instead of the normal
fifo, added a ")" and a newline at the end of each, and it imported just
fine!

INSERT INTO logs (host, facility, priority, level, tag, date, time,
program, msg) VALUES ( 'X.X.X.X', 'local4', 'notice', 'notice', 'a5',
'2004-11-19', '16:48:04', 'de-gw-1', 'de-gw-1: NetScreen
device_id=3Dde-gw-1  [No Name]system-notification-00257(traffic):
start_time=3D\"2004-11-19 18:02:12\" duration=3D3 policy_id=3D2
service=3DNETBIOS (SSN) proto=3D6 src zone=3DTrust dst zone=3DUntrust
action=3DPermit sent=3D1813 rcvd=3D5846 src=3DX.X.X.X dst=3DX.X.X.X =
src_port=3D3525
dst_port=3D139 src-xlated ip=3DX.X.X.X port=3D3525 INSERT INTO logs =
(host,
facility, priority, level, tag, date, time, program, msg) VALUES (
'X.X.X.X', 'local4', 'notice', 'notice', 'a5', '2004-11-19', '16:48:06',
'de-gw-1', 'de-gw-1: NetScreen device_id=3Dde-gw-1  [No
Name]system-notification-00257(traffic): start_time=3D\"2004-11-19
18:02:14\" duration=3D3 policy_id=3D3 service=3Dicmp proto=3D1 src =
zone=3DUntrust
dst zone=3DTrust action=3DPermit sent=3D78 rcvd=3D78 src=3DX.X.X.X =
dst=3DX.X.X.X
icmp type=3D8 INSERT INTO logs (host, facility, priority, level, tag,
date, time, program, msg) VALUES ( 'X.X.X.X', 'local4', 'notice',
'notice', 'a5', '2004-11-19', '16:48:06', 'de-gw-1', 'de-gw-1: NetScreen
device_id=3Dde-gw-1  [No Name]system-notification-00257(traffic):
start_time=3D\"2004-11-19 18:01:15\" duration=3D62 policy_id=3D3
service=3Dudp/port:3077 proto=3D17 src zone=3DUntrust dst zone=3DTrust
action=3DPermit sent=3D0 rcvd=3D0 src=3DX.X.X.X dst=3DX.X.X.X =
src_port=3D17359
dst_port=3D3077 src-xlated ip=3DX.X.X.X port=3D17359=20

Remember that this is happening just for the Netscreen devices, using
identical syslog-ng configs as for the other device types.

Here's an example config:

source net { udp(); };
filter f_firewalls { facility(local4); };
log { source(net); filter(f_firewalls); destination(firewalls); };

destination firewalls { pipe("/tmp/mysql_firewalls.pipe" template("
INSERT INTO logs (host, facility, priority, level, tag, date, time,
program, msg) VALUES ( '$HOST', '$FACILITY', '$PRIORITY', '$LEVEL',
'$TAG', '$YEAR-$MONTH-$DAY', '$HOUR:$MIN:$SEC', '$PROGRAM', '$MSG
');\n") template-escape(yes));
};

This is on Gentoo Linux using Syslog-ng 1.6.4

stats(43200);
log_msg_size(8192);
log_fifo_size(8192);

As per http://www.campin.net/syslog-ng/faq.html#message_length - is this
the same issue?

I could do some obscure perl filtering to fix this on the fly, but I'd
rather fix the problem rather than the symptom.

Thanks for your time.

James Masson