[syslog-ng] Debugging Pattern Match Failures

[Martin Holste]

No worries, happy to help.

Ok, try this destination:

template t_test { template("$MSGONLY\n"); };

destination d_test {
  file("/logs/raw/test" dir_owner("root") owner("root") group("root")
perm(0640) dir_perm(0755) create_dirs(yes)
    template(t_test)
  );
};

And don't use the "r_raw" rewriter with this destination.  I'm pretty
sure this will yield what you've got already, but I want to take as
many variables out of it as possible.

If the messages do indeed look the same in /logs/raw/test, then it
means that you need to write your patterns so that they start matching
from whatever is printed.  That shouldn't be too bad as long as you
can get a decent delimiter in there.  The colons look like a good
anchor.

On Tue, Aug 3, 2010 at 8:35 PM, Matthew Hall <mhall at mhcomputing.net> wrote:
> I did some experimentation, using the following log setup:
>
> rewrite r_raw {
>    set("$MSGONLY");
> };
>
> destination d_u_raw_local1 {
>    file("/logs/raw/local1"
>         dir_owner("root")
>         owner("root")
>         group("root")
>         perm(0640)
>         dir_perm(0755)
>         create_dirs(yes)
>         template(t_default) suppress(3)
>    );
> };
>
> But I am still getting messages like this:
>
> Aug 1 00:00:00 <local1.notice> 172.16.0.2 from: 172.16.0.1: 000001: Aug 1 00:00:00.000: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5/16, changed state to up
>
> This it seems that I am not successfully stripping all headers normally
> added by the file writer off of the message using this configuration.
> What did I miss here in my rewrite rule? Without some way to make sure I
> have a raw file with no weird headers added it's hard to make decent
> patterns.
>
> Thanks,
> Matthew.
>
> On Tue, Aug 03, 2010 at 05:18:10PM -0700, Matthew Hall wrote:
>> On Tue, Aug 03, 2010 at 06:53:13PM -0500, Martin Holste wrote:
>> > I believe the matching is done against the $MSGONLY macro, so you can
>> > put another log destination in to write that out only and have a look
>> > to see what the parser is seeing.  Do you have an example log you can
>> > show?
>>
>> Here is an example of what would be appearing in the disk log file:
>>
>> Jul  1 00:00:00 <local1.notice> 172.16.0.1 0000001: Jul  1 00:00:00.000 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
>>
>> There are many more types of message coming from many more devices, some
>> of which are BSD compliant and some of which are not, and I think that
>> is part of my problem.
>>
>> The unclear part is how much of the front part needs to be stripped off,
>> before making the patterns in the XML file. Hopefully I will be able to
>> figure that out now that you have clarified how I can make a raw message
>> file without extraneous strings appended.
>>
>> Thanks for helping me understand how this works and what I can do to get
>> my patterns right. I definitely owe you a beer.
>>
>> Regards,
>> Matthew.
>>
>> > On Tue, Aug 3, 2010 at 12:10 PM, Matthew Hall <mhall at mhcomputing.net> wrote:
>> > > On Tue, Aug 03, 2010 at 02:39:38PM +0200, Balazs Scheidler wrote:
>> > >> Well, if you want to look at the result of the message parsing exactly
>> > >> as done by syslog-ng, you could use a noop rewrite rule and enable
>> > >> debugging (though it is not recommended to be done in a production
>> > >> server):
>> > >>
>> > >> rewrite r_noop { set("$MESSAGE"); };
>> > >>
>> > >> This would set $MESSAGE to $MESSAGE, but at the end of the rewrite rule,
>> > >> syslog-ng would emit a debug message about the contents of the MESSAGE
>> > >> name-value pair.
>> > >
>> > > Unfortunately I can't even get that far because the beginning of my
>> > > message patterns is not matching up against whatever syslog-ng is using
>> > > to do the pattern match, so I am not going to get any name value pairs
>> > > out.
>> > >
>> > >> Alternatively, you may still be able to use "pdbtool match" which can
>> > >> read a log file, parse it with syslog-ng's message parser and report the
>> > >> results per name-value pair.
>> > >>
>> > >> $ pdbtool match -f /var/log/auth.log -p access/sshd.pdb  | head -10
>> > >> HOST=bzorp
>> > >> MESSAGE=pam_unix(cron:session): session opened for user root by (uid=0)
>> > >> PROGRAM=CRON
>> > >> PID=7362
>> > >> LEGACY_MSGHDR=CRON[7362]:
>> > >> .classifier.class=unknown
>> > >>
>> > >> ...
>> > >>
>> > >> This uses the normal BSD syslog parser to read the file (thus if you are
>> > >> using no-parse flag, or RFC5424 format log files, that may differ)
>> > >
>> > > How do I create a file in this BSD format the pdbtool expects? Right now
>> > > I am using syslog-ng output files as input to my patternizing scripts,
>> > > but I think I am not stripping off the right things at the beginning of
>> > > the lines in these files (either too much or too little).
>> > >
>> > > Is there some option I can use to store just the part it would send to
>> > > the pattern matcher so that I can have input to my patternizer which
>> > > looks exactly like what the daemon is going to match during the pattern
>> > > match for each message?
>> > >
>> > >> --
>> > >> Bazsi
>> > >
>> > > Thanks,
>> > > Matthew.
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>