[syslog-ng] Re-post: What testing has been done for the new features in open source version of syslog-ng version 3.x

[Balazs Scheidler]

hi,

On Tue, 2010-02-02 at 08:36 -0600, Jeremy M. Guthrie wrote:
> Sorry, I didn't see an answer.  Just wanted to re-post to reconfirm:
> 
> I have an issue where what is being written to file is different than what I 
> have being written to disk.  I don't know if the macro is broken, my 
> understanding is wrong, or I am just using the wrong macro altogether.

I guess you meant here that file/SQL is different, right?

> 
> My understanding is roughly as follows:
> PROGRAM	The name of the program sending the message.
> PID	The PID of the program sending the message.
> MSG or MESSAGE	Message contents including the program name and pid.
> MSGONLY	Message contents without the program name. 

this is true for 2.0 and 2.1, syslog-ng 3.0 changed this behaviour and
although it can operate in 2.1 compatibility mode, it all depends on the
syslog-ng.conf file, whether it contains '@version: 3.0' at the front.

If there's no @version directive, or it contains '2.1', the things you
wrote above is true. If you are running syslog-ng in 3.0 mode (which
performs better), then

$MSG: doesn't contain the program name and pid any more, basically it is
the same as $MSGONLY
$MSGHDR: contains the combined program/pid value

and of course you do have the separate macros ($PROGRAM, $PID)

> 
> 1.  The first thing I ran into:
> Example messages on file:
> Jan 25 08:34:15 XXXXXXXXXX sshd[7209]: service name is :: sshd
> 
> Same example message in SQL:
> Jan 25 08:34:15 XXXXXXXXXX service name is :: sshd
> 
> template("INSERT INTO tsyslog (host, facility, severity, date, time, message) 
> VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN:
> $SEC', '$MSG');\n" )

did you specify a template for your file? if you didn't then this is
correct 3.0  behaviour.

> 
> 2.  Then what I tried to get it closer....
> Example messages on file:
> Jan 25 08:40:18 XXXXXXXXXX sshd[7209]: service name is :: sshd
> 
> Same example message in SQL:
> Jan 25 08:40:18 XXXXXXXXXX sshd: service name is :: sshd
> 
> template("INSERT INTO tsyslog (host, facility, severity, date, time, message) 
> VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN:
> $SEC', '$PROGRAM: $MSG');\n" )

again. this is correct.

> 
> 3.  Then what I tried to get even closer:
> Example messages on file:
> Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd
> 
> Same example message in SQL:
> Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd
> 
> template("INSERT INTO tsyslog (host, facility, severity, date, time, message) 
> VALUES ( '$HOST','$FACILITY_NUM','$LEVEL_NUM','$YEAR-$MONTH-$DAY','$HOUR:$MIN:
> $SEC', '$PROGRAM[$PID]: $MSG');\n" )

correct again.

> 
> However, this broke messages coming from Cisco devices...
> Jan 25 08:55:10 FIREWALL %FWSM-3-710003[]: tcp access denied by ACL from 
> 188.23.180.240/3826 to outside:172.26.54.65/445
> 
> The message above should be:
> Jan 25 08:55:10 FIREWALL %FWSM-3-710003: tcp access denied by ACL from 
> 188.23.180.240/3826 to outside:172.26.54.65/445
> 
> Notice the above '[]' after the FWSM code?  I am running 3.0.5 syslog-ng.  
> 
> Ideally I'd get both:
> Jan 25 08:55:10 FIREWALL %FWSM-3-710003: tcp access denied by ACL from 
> 188.23.180.240/3826 to outside:172.26.54.65/445
> 
> AND
> 
> Jan 25 08:53:15 XXXXXXXXXX sshd[7209]: service name is :: sshd
> 
> Is the MSG/MESSAGE macro broken?  What am I missing?

you could perhaps use the $MSGHDR macro, which combines the program/pid
fields and omits the '[]' part if it is not needed.

Also you could possibly need the 'store-legacy-msghdr' macro on your
input to make it absolutely sure that the log message remains intact.
(without this flag, syslog-ng is trying to parse/reconstruct the
program[pid] part, which may be slightly different than the original
input).


-- 
Bazsi