[syslog-ng] Debugging Pattern Match Failures

[Matthew Hall]

On Tue, Aug 03, 2010 at 06:53:13PM -0500, Martin Holste wrote:
> I believe the matching is done against the $MSGONLY macro, so you can
> put another log destination in to write that out only and have a look
> to see what the parser is seeing.  Do you have an example log you can
> show?

Here is an example of what would be appearing in the disk log file:

Jul  1 00:00:00 <local1.notice> 172.16.0.1 0000001: Jul  1 00:00:00.000 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down

There are many more types of message coming from many more devices, some 
of which are BSD compliant and some of which are not, and I think that 
is part of my problem.

The unclear part is how much of the front part needs to be stripped off, 
before making the patterns in the XML file. Hopefully I will be able to 
figure that out now that you have clarified how I can make a raw message 
file without extraneous strings appended.

Thanks for helping me understand how this works and what I can do to get 
my patterns right. I definitely owe you a beer.

Regards,
Matthew.

> On Tue, Aug 3, 2010 at 12:10 PM, Matthew Hall <mhall at mhcomputing.net> wrote:
> > On Tue, Aug 03, 2010 at 02:39:38PM +0200, Balazs Scheidler wrote:
> >> Well, if you want to look at the result of the message parsing exactly
> >> as done by syslog-ng, you could use a noop rewrite rule and enable
> >> debugging (though it is not recommended to be done in a production
> >> server):
> >>
> >> rewrite r_noop { set("$MESSAGE"); };
> >>
> >> This would set $MESSAGE to $MESSAGE, but at the end of the rewrite rule,
> >> syslog-ng would emit a debug message about the contents of the MESSAGE
> >> name-value pair.
> >
> > Unfortunately I can't even get that far because the beginning of my
> > message patterns is not matching up against whatever syslog-ng is using
> > to do the pattern match, so I am not going to get any name value pairs
> > out.
> >
> >> Alternatively, you may still be able to use "pdbtool match" which can
> >> read a log file, parse it with syslog-ng's message parser and report the
> >> results per name-value pair.
> >>
> >> $ pdbtool match -f /var/log/auth.log -p access/sshd.pdb  | head -10
> >> HOST=bzorp
> >> MESSAGE=pam_unix(cron:session): session opened for user root by (uid=0)
> >> PROGRAM=CRON
> >> PID=7362
> >> LEGACY_MSGHDR=CRON[7362]:
> >> .classifier.class=unknown
> >>
> >> ...
> >>
> >> This uses the normal BSD syslog parser to read the file (thus if you are
> >> using no-parse flag, or RFC5424 format log files, that may differ)
> >
> > How do I create a file in this BSD format the pdbtool expects? Right now
> > I am using syslog-ng output files as input to my patternizing scripts,
> > but I think I am not stripping off the right things at the beginning of
> > the lines in these files (either too much or too little).
> >
> > Is there some option I can use to store just the part it would send to
> > the pattern matcher so that I can have input to my patternizer which
> > looks exactly like what the daemon is going to match during the pattern
> > match for each message?
> >
> >> --
> >> Bazsi
> >
> > Thanks,
> > Matthew.