No worries, happy to help. Ok, try this destination: template t_test { template("$MSGONLY\n"); }; destination d_test { file("/logs/raw/test" dir_owner("root") owner("root") group("root") perm(0640) dir_perm(0755) create_dirs(yes) template(t_test) ); }; And don't use the "r_raw" rewriter with this destination. I'm pretty sure this will yield what you've got already, but I want to take as many variables out of it as possible. If the messages do indeed look the same in /logs/raw/test, then it means that you need to write your patterns so that they start matching from whatever is printed. That shouldn't be too bad as long as you can get a decent delimiter in there. The colons look like a good anchor. On Tue, Aug 3, 2010 at 8:35 PM, Matthew Hall <mhall@mhcomputing.net> wrote:
I did some experimentation, using the following log setup:
rewrite r_raw { set("$MSGONLY"); };
destination d_u_raw_local1 { file("/logs/raw/local1" dir_owner("root") owner("root") group("root") perm(0640) dir_perm(0755) create_dirs(yes) template(t_default) suppress(3) ); };
But I am still getting messages like this:
Aug 1 00:00:00 <local1.notice> 172.16.0.2 from: 172.16.0.1: 000001: Aug 1 00:00:00.000: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5/16, changed state to up
This it seems that I am not successfully stripping all headers normally added by the file writer off of the message using this configuration. What did I miss here in my rewrite rule? Without some way to make sure I have a raw file with no weird headers added it's hard to make decent patterns.
Thanks, Matthew.
On Tue, Aug 03, 2010 at 05:18:10PM -0700, Matthew Hall wrote:
On Tue, Aug 03, 2010 at 06:53:13PM -0500, Martin Holste wrote:
I believe the matching is done against the $MSGONLY macro, so you can put another log destination in to write that out only and have a look to see what the parser is seeing. Do you have an example log you can show?
Here is an example of what would be appearing in the disk log file:
Jul 1 00:00:00 <local1.notice> 172.16.0.1 0000001: Jul 1 00:00:00.000 UTC: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/1, changed state to down
There are many more types of message coming from many more devices, some of which are BSD compliant and some of which are not, and I think that is part of my problem.
The unclear part is how much of the front part needs to be stripped off, before making the patterns in the XML file. Hopefully I will be able to figure that out now that you have clarified how I can make a raw message file without extraneous strings appended.
Thanks for helping me understand how this works and what I can do to get my patterns right. I definitely owe you a beer.
Regards, Matthew.
On Tue, Aug 3, 2010 at 12:10 PM, Matthew Hall <mhall@mhcomputing.net> wrote:
On Tue, Aug 03, 2010 at 02:39:38PM +0200, Balazs Scheidler wrote:
Well, if you want to look at the result of the message parsing exactly as done by syslog-ng, you could use a noop rewrite rule and enable debugging (though it is not recommended to be done in a production server):
rewrite r_noop { set("$MESSAGE"); };
This would set $MESSAGE to $MESSAGE, but at the end of the rewrite rule, syslog-ng would emit a debug message about the contents of the MESSAGE name-value pair.
Unfortunately I can't even get that far because the beginning of my message patterns is not matching up against whatever syslog-ng is using to do the pattern match, so I am not going to get any name value pairs out.
Alternatively, you may still be able to use "pdbtool match" which can read a log file, parse it with syslog-ng's message parser and report the results per name-value pair.
$ pdbtool match -f /var/log/auth.log -p access/sshd.pdb | head -10 HOST=bzorp MESSAGE=pam_unix(cron:session): session opened for user root by (uid=0) PROGRAM=CRON PID=7362 LEGACY_MSGHDR=CRON[7362]: .classifier.class=unknown
...
This uses the normal BSD syslog parser to read the file (thus if you are using no-parse flag, or RFC5424 format log files, that may differ)
How do I create a file in this BSD format the pdbtool expects? Right now I am using syslog-ng output files as input to my patternizing scripts, but I think I am not stripping off the right things at the beginning of the lines in these files (either too much or too little).
Is there some option I can use to store just the part it would send to the pattern matcher so that I can have input to my patternizer which looks exactly like what the daemon is going to match during the pattern match for each message?
-- Bazsi
Thanks, Matthew.
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html