[syslog-ng] Debugging Pattern Match Failures

Martin Holste mcholste at gmail.com
Wed Aug 4 05:18:14 CEST 2010 

Ok, this pattern seems to work in the pdbtest tool:

<ruleset>
		<rules>
			<rule class="test" id="test">
				<patterns>
					<pattern>@ESTRING:month: @@NUMBER:mday:@
@NUMBER:hour:@:@NUMBER:minute:@:@NUMBER:second:@
@QSTRING:priority:&lt;&gt;@ @IPv4:host1:@ from: @IPv4:host2:@:
@NUMBER::@: @ESTRING:: @@NUMBER::@
@NUMBER::@:@NUMBER::@:@NUMBER::@. at NUMBER::@: @ESTRING:program::@
@ANYSTRING:message:@</pattern>
				</patterns>
				<examples>
					<example>
						<test_message program="">Aug 1 00:00:00 &lt;local1.notice&gt;
172.16.0.2 from: 172.16.0.1: 000001: Aug 1 00:00:00.000:
%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5/16,
changed state to up</test_message>
						<test_values>
							<test_value name="month">Aug</test_value>
							<test_value name="mday">1</test_value>
							<test_value name="hour">00</test_value>
							<test_value name="minute">00</test_value>
							<test_value name="second">00</test_value>
							<test_value name="priority">local1.notice</test_value>
							<test_value name="host1">172.16.0.2</test_value>
							<test_value name="host2">172.16.0.1</test_value>
							<test_value name="program">%LINEPROTO-5-UPDOWN</test_value>
							<test_value name="message">Line protocol on Interface
FastEthernet5/16, changed state to up</test_value>
						</test_values>
					</example>
				</examples>
			</rule>
		</rules>
	</ruleset>

So give that a shot with the tool and then it's just a matter of
getting your $MSGONLY macro to line up.

On Tue, Aug 3, 2010 at 9:11 PM, Martin Holste <mcholste at gmail.com> wrote:
> No worries, happy to help.
>
> Ok, try this destination:
>
> template t_test { template("$MSGONLY\n"); };
>
> destination d_test {
>  file("/logs/raw/test" dir_owner("root") owner("root") group("root")
> perm(0640) dir_perm(0755) create_dirs(yes)
>    template(t_test)
>  );
> };
>
> And don't use the "r_raw" rewriter with this destination.  I'm pretty
> sure this will yield what you've got already, but I want to take as
> many variables out of it as possible.
>
> If the messages do indeed look the same in /logs/raw/test, then it
> means that you need to write your patterns so that they start matching
> from whatever is printed.  That shouldn't be too bad as long as you
> can get a decent delimiter in there.  The colons look like a good
> anchor.
>
> On Tue, Aug 3, 2010 at 8:35 PM, Matthew Hall <mhall at mhcomputing.net> wrote:
>> I did some experimentation, using the following log setup:
>>
>> rewrite r_raw {
>>    set("$MSGONLY");
>> };
>>
>> destination d_u_raw_local1 {
>>    file("/logs/raw/local1"
>>         dir_owner("root")
>>         owner("root")
>>         group("root")
>>         perm(0640)
>>         dir_perm(0755)
>>         create_dirs(yes)
>>         template(t_default) suppress(3)
>>    );
>> };
>>
>> But I am still getting messages like this:
>>
>> Aug 1 00:00:00 <local1.notice> 172.16.0.2 from: 172.16.0.1: 000001: Aug 1 00:00:00.000: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5/16, changed state to up
>>
>> This it seems that I am not successfully stripping all headers normally
>> added by the file writer off of the message using this configuration.
>> What did I miss here in my rewrite rule? Without some way to make sure I
>> have a raw file with no weird headers added it's hard to make decent
>> patterns.
>>
>> Thanks,
>> Matthew.
>>
>> On Tue, Aug 03, 2010 at 05:18:10PM -0700, Matthew Hall wrote:
>>> On Tue, Aug 03, 2010 at 06:53:13PM -0500, Martin Holste wrote:
>>> > I believe the matching is done against the $MSGONLY macro, so you can
>>> > put another log destination in to write that out only and have a look
>>> > to see what the parser is seeing.  Do you have an example log you can
>>> > show?
>>>
>>> Here is an example of what would be appearing in the disk log file:
>>>
>>> Jul  1 00:00:00 <local1.notice> 172.16.0.1 0000001: Jul  1 00:00:00.000 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
>>>
>>> There are many more types of message coming from many more devices, some
>>> of which are BSD compliant and some of which are not, and I think that
>>> is part of my problem.
>>>
>>> The unclear part is how much of the front part needs to be stripped off,
>>> before making the patterns in the XML file. Hopefully I will be able to
>>> figure that out now that you have clarified how I can make a raw message
>>> file without extraneous strings appended.
>>>
>>> Thanks for helping me understand how this works and what I can do to get
>>> my patterns right. I definitely owe you a beer.
>>>
>>> Regards,
>>> Matthew.
>>>
>>> > On Tue, Aug 3, 2010 at 12:10 PM, Matthew Hall <mhall at mhcomputing.net> wrote:
>>> > > On Tue, Aug 03, 2010 at 02:39:38PM +0200, Balazs Scheidler wrote:
>>> > >> Well, if you want to look at the result of the message parsing exactly
>>> > >> as done by syslog-ng, you could use a noop rewrite rule and enable
>>> > >> debugging (though it is not recommended to be done in a production
>>> > >> server):
>>> > >>
>>> > >> rewrite r_noop { set("$MESSAGE"); };
>>> > >>
>>> > >> This would set $MESSAGE to $MESSAGE, but at the end of the rewrite rule,
>>> > >> syslog-ng would emit a debug message about the contents of the MESSAGE
>>> > >> name-value pair.
>>> > >
>>> > > Unfortunately I can't even get that far because the beginning of my
>>> > > message patterns is not matching up against whatever syslog-ng is using
>>> > > to do the pattern match, so I am not going to get any name value pairs
>>> > > out.
>>> > >
>>> > >> Alternatively, you may still be able to use "pdbtool match" which can
>>> > >> read a log file, parse it with syslog-ng's message parser and report the
>>> > >> results per name-value pair.
>>> > >>
>>> > >> $ pdbtool match -f /var/log/auth.log -p access/sshd.pdb  | head -10
>>> > >> HOST=bzorp
>>> > >> MESSAGE=pam_unix(cron:session): session opened for user root by (uid=0)
>>> > >> PROGRAM=CRON
>>> > >> PID=7362
>>> > >> LEGACY_MSGHDR=CRON[7362]:
>>> > >> .classifier.class=unknown
>>> > >>
>>> > >> ...
>>> > >>
>>> > >> This uses the normal BSD syslog parser to read the file (thus if you are
>>> > >> using no-parse flag, or RFC5424 format log files, that may differ)
>>> > >
>>> > > How do I create a file in this BSD format the pdbtool expects? Right now
>>> > > I am using syslog-ng output files as input to my patternizing scripts,
>>> > > but I think I am not stripping off the right things at the beginning of
>>> > > the lines in these files (either too much or too little).
>>> > >
>>> > > Is there some option I can use to store just the part it would send to
>>> > > the pattern matcher so that I can have input to my patternizer which
>>> > > looks exactly like what the daemon is going to match during the pattern
>>> > > match for each message?
>>> > >
>>> > >> --
>>> > >> Bazsi
>>> > >
>>> > > Thanks,
>>> > > Matthew.
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.campin.net/syslog-ng/faq.html
>>
>>
>

More information about the syslog-ng mailing list