[syslog-ng] Debugging Pattern Match Failures

Wed Aug 4 23:07:49 CEST 2010

Brilliant advice Martin. It seems that this new output template has 
indeed resolved my issue. I will collect some new log files and rerun my 
patternizer.

172.16.0.1: 000001: Aug  1 00:00:00.000: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up

Best Regards,
Matthew.

On Tue, Aug 03, 2010 at 09:11:08PM -0500, Martin Holste wrote:
> No worries, happy to help.
> 
> Ok, try this destination:
> 
> template t_test { template("$MSGONLY\n"); };
> 
> destination d_test {
>   file("/logs/raw/test" dir_owner("root") owner("root") group("root")
> perm(0640) dir_perm(0755) create_dirs(yes)
>     template(t_test)
>   );
> };
> 
> And don't use the "r_raw" rewriter with this destination.  I'm pretty
> sure this will yield what you've got already, but I want to take as
> many variables out of it as possible.
> 
> If the messages do indeed look the same in /logs/raw/test, then it
> means that you need to write your patterns so that they start matching
> from whatever is printed.  That shouldn't be too bad as long as you
> can get a decent delimiter in there.  The colons look like a good
> anchor.
> 
> On Tue, Aug 3, 2010 at 8:35 PM, Matthew Hall <mhall at mhcomputing.net> wrote:
> > I did some experimentation, using the following log setup:
> >
> > rewrite r_raw {
> >    set("$MSGONLY");
> > };
> >
> > destination d_u_raw_local1 {
> >    file("/logs/raw/local1"
> >         dir_owner("root")
> >         owner("root")
> >         group("root")
> >         perm(0640)
> >         dir_perm(0755)
> >         create_dirs(yes)
> >         template(t_default) suppress(3)
> >    );
> > };
> >
> > But I am still getting messages like this:
> >
> > Aug 1 00:00:00 <local1.notice> 172.16.0.2 from: 172.16.0.1: 000001: Aug 1 00:00:00.000: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5/16, changed state to up
> >
> > This it seems that I am not successfully stripping all headers normally
> > added by the file writer off of the message using this configuration.
> > What did I miss here in my rewrite rule? Without some way to make sure I
> > have a raw file with no weird headers added it's hard to make decent
> > patterns.
> >
> > Thanks,
> > Matthew.
> >
> > On Tue, Aug 03, 2010 at 05:18:10PM -0700, Matthew Hall wrote:
> >> On Tue, Aug 03, 2010 at 06:53:13PM -0500, Martin Holste wrote:
> >> > I believe the matching is done against the $MSGONLY macro, so you can
> >> > put another log destination in to write that out only and have a look
> >> > to see what the parser is seeing.  Do you have an example log you can
> >> > show?
> >>
> >> Here is an example of what would be appearing in the disk log file:
> >>
> >> Jul  1 00:00:00 <local1.notice> 172.16.0.1 0000001: Jul  1 00:00:00.000 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
> >>
> >> There are many more types of message coming from many more devices, some
> >> of which are BSD compliant and some of which are not, and I think that
> >> is part of my problem.
> >>
> >> The unclear part is how much of the front part needs to be stripped off,
> >> before making the patterns in the XML file. Hopefully I will be able to
> >> figure that out now that you have clarified how I can make a raw message
> >> file without extraneous strings appended.
> >>
> >> Thanks for helping me understand how this works and what I can do to get
> >> my patterns right. I definitely owe you a beer.
> >>
> >> Regards,
> >> Matthew.
> >>
> >> > On Tue, Aug 3, 2010 at 12:10 PM, Matthew Hall <mhall at mhcomputing.net> wrote:
> >> > > On Tue, Aug 03, 2010 at 02:39:38PM +0200, Balazs Scheidler wrote:
> >> > >> Well, if you want to look at the result of the message parsing exactly
> >> > >> as done by syslog-ng, you could use a noop rewrite rule and enable
> >> > >> debugging (though it is not recommended to be done in a production
> >> > >> server):
> >> > >>
> >> > >> rewrite r_noop { set("$MESSAGE"); };
> >> > >>
> >> > >> This would set $MESSAGE to $MESSAGE, but at the end of the rewrite rule,
> >> > >> syslog-ng would emit a debug message about the contents of the MESSAGE
> >> > >> name-value pair.
> >> > >
> >> > > Unfortunately I can't even get that far because the beginning of my
> >> > > message patterns is not matching up against whatever syslog-ng is using
> >> > > to do the pattern match, so I am not going to get any name value pairs
> >> > > out.
> >> > >
> >> > >> Alternatively, you may still be able to use "pdbtool match" which can
> >> > >> read a log file, parse it with syslog-ng's message parser and report the
> >> > >> results per name-value pair.
> >> > >>
> >> > >> $ pdbtool match -f /var/log/auth.log -p access/sshd.pdb  | head -10
> >> > >> HOST=bzorp
> >> > >> MESSAGE=pam_unix(cron:session): session opened for user root by (uid=0)
> >> > >> PROGRAM=CRON
> >> > >> PID=7362
> >> > >> LEGACY_MSGHDR=CRON[7362]:
> >> > >> .classifier.class=unknown
> >> > >>
> >> > >> ...
> >> > >>
> >> > >> This uses the normal BSD syslog parser to read the file (thus if you are
> >> > >> using no-parse flag, or RFC5424 format log files, that may differ)
> >> > >
> >> > > How do I create a file in this BSD format the pdbtool expects? Right now
> >> > > I am using syslog-ng output files as input to my patternizing scripts,
> >> > > but I think I am not stripping off the right things at the beginning of
> >> > > the lines in these files (either too much or too little).
> >> > >
> >> > > Is there some option I can use to store just the part it would send to
> >> > > the pattern matcher so that I can have input to my patternizer which
> >> > > looks exactly like what the daemon is going to match during the pattern
> >> > > match for each message?
> >> > >
> >> > >> --
> >> > >> Bazsi
> >> > >
> >> > > Thanks,
> >> > > Matthew.
> > ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.campin.net/syslog-ng/faq.html
> >
> >
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
> 