Re: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there. how to do on this ----->>>> If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work. 2013/4/28 <syslog-ng-request@lists.balabit.hu>
Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu
To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu
You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu
When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (????) 2. Re: Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (Evan Rempel)
----------------------------------------------------------------------
Message: 1 Date: Sat, 27 Apr 2013 22:34:50 +0800 From: ???? <onlydebian@gmail.com> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) To: syslog-ng@lists.balabit.hu Message-ID: <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR= V7wJ1EJdvE6Zx9zg@mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1"
when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time}
test log output ,just like this. === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP ,
do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot.
root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE" Pattern matching part: @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@ @ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ Matching part: Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE Values: MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE PROGRAM=ESXI .classifier.class=esxi .classifier.rule_id=182437592347598 .esxi.month=Apr .esxi.date=26 .esxi.time=15:17:31 .esxi.host_ip=192.168.88.71 .esxi.program= vmkernel .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE root@debian:~#
my configuration like as below
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>ESXI</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
######## syslog-ng.conf ########
#####Parser##### parser pattern_db { db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); };
#Check pattern matching destination udp_esxi_output { file("/var/log/pattern_output" template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n") template_escape(no)); };
#####Log##### log { source(s_network); parser(pattern_db); destination(udp_esxi_output); };
Sorry for not being more clear in my first response. You have a template of template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP ${.esxi.host_ip},${.esxi.message}\n") When syslog-ng receives a syslog message, it logged it as; === system,error,critical, HOST IP , This means that $PROGRAM contains "system" Now for the patterndb part. The patterndb parser FIRST matches $PROGRAM To the <pattern>XXXX</pattern> in the <ruleset> <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>XXXX</pattern> In your case you have specified <pattern>ESXI</pattern> so the patterndb parser will NOT use any of your patterndb because it does not match the $PROGRAM You need to use ######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>system</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> You have not included a complete syslong-ng source line for me to see what you are trying to match against so I can not tell if you pattern will actually match the lines that you are trying to match. At my organization we run ESX as well, and none of our lines would match the pattern that you have, but your environment might be different. I hope this was more clear. Evan. ________________________________________ From: 不坏阿峰 [onlydebian@gmail.com] Sent: Sunday, April 28, 2013 8:24 AM To: syslog-ng@lists.balabit.hu; Evan Rempel Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there. how to do on this ----->>>> If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work. 2013/4/28 <syslog-ng-request@lists.balabit.hu<mailto:syslog-ng-request@lists.balabit.hu>> Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu<mailto:syslog-ng-request@lists.balabit.hu> You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu<mailto:syslog-ng-owner@lists.balabit.hu> When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (????) 2. Re: Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (Evan Rempel) ---------------------------------------------------------------------- Message: 1 Date: Sat, 27 Apr 2013 22:34:50 +0800 From: ???? <onlydebian@gmail.com<mailto:onlydebian@gmail.com>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Message-ID: <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR=V7wJ1EJdvE6Zx9zg@mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com>> Content-Type: text/plain; charset="iso-8859-1" when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time} test log output ,just like this. === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot. root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE" Pattern matching part: @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ Matching part: Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE Values: MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE PROGRAM=ESXI .classifier.class=esxi .classifier.rule_id=182437592347598 .esxi.month=Apr .esxi.date=26 .esxi.time=15:17:31 .esxi.host_ip=192.168.88.71 .esxi.program= vmkernel .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE root@debian:~# my configuration like as below ######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>ESXI</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> ######## syslog-ng.conf ######## #####Parser##### parser pattern_db { db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); }; #Check pattern matching destination udp_esxi_output { file("/var/log/pattern_output" template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n") template_escape(no)); }; #####Log##### log { source(s_network); parser(pattern_db); destination(udp_esxi_output); };
attachment is my current syslog-ng.conf. and esxi_pattern.xml. my syslog-ng receive UDP log from esxi host and try to test the db-parse and log it. i have change to <pattern>system</pattern>, but still can not get value from parse refer macro. thanks. 2013/4/28 Evan Rempel <erempel@uvic.ca>
Sorry for not being more clear in my first response.
You have a template of
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP ${.esxi.host_ip},${.esxi.message}\n")
When syslog-ng receives a syslog message, it logged it as;
=== system,error,critical, HOST IP ,
This means that $PROGRAM contains "system"
Now for the patterndb part.
The patterndb parser FIRST matches $PROGRAM To the <pattern>XXXX</pattern> in the <ruleset>
<?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>XXXX</pattern>
In your case you have specified <pattern>ESXI</pattern> so the patterndb parser will NOT use any of your patterndb because it does not match the $PROGRAM
You need to use
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>system</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
You have not included a complete syslong-ng source line for me to see what you are trying to match against so I can not tell if you pattern will actually match the lines that you are trying to match. At my organization we run ESX as well, and none of our lines would match the pattern that you have, but your environment might be different.
I hope this was more clear.
Evan.
________________________________________ From: 不坏阿峰 [onlydebian@gmail.com] Sent: Sunday, April 28, 2013 8:24 AM To: syslog-ng@lists.balabit.hu; Evan Rempel Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there.
how to do on this ----->>>> If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work.
2013/4/28 <syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>> Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>
To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>
You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu<mailto: syslog-ng-owner@lists.balabit.hu>
When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (????) 2. Re: Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (Evan Rempel)
----------------------------------------------------------------------
Message: 1 Date: Sat, 27 Apr 2013 22:34:50 +0800 From: ???? <onlydebian@gmail.com<mailto:onlydebian@gmail.com>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Message-ID: <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR= V7wJ1EJdvE6Zx9zg@mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com>> Content-Type: text/plain; charset="iso-8859-1"
when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time}
test log output ,just like this. === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP ,
do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot.
root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE" Pattern matching part: @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@ @ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ Matching part: Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE Values: MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE PROGRAM=ESXI .classifier.class=esxi .classifier.rule_id=182437592347598 .esxi.month=Apr .esxi.date=26 .esxi.time=15:17:31 .esxi.host_ip=192.168.88.71 .esxi.program= vmkernel .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE root@debian:~#
my configuration like as below
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>ESXI</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
######## syslog-ng.conf ########
#####Parser##### parser pattern_db { db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); };
#Check pattern matching destination udp_esxi_output { file("/var/log/pattern_output" template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n") template_escape(no)); };
#####Log##### log { source(s_network); parser(pattern_db); destination(udp_esxi_output); };
sorry for miss the purpose what i want to do. (1) first . receive syslog from esxi host from UDP . (done) (2) second. parse the log from UDP and parse with pattern db and get separate imformation ( meet the problem i ask for help) (3)third. store separate infor to Oracle table(done, test successfully on syslog-ng macro value) for the second step, i use the way file() to check the situation of db-parse. some sample log message from esxi host. Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP: mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0) Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default'] Unrecognized 2013/4/29 不坏阿峰 <onlydebian@gmail.com>
attachment is my current syslog-ng.conf. and esxi_pattern.xml.
my syslog-ng receive UDP log from esxi host and try to test the db-parse and log it.
i have change to <pattern>system</pattern>, but still can not get value from parse refer macro.
thanks.
2013/4/28 Evan Rempel <erempel@uvic.ca>
Sorry for not being more clear in my first response.
You have a template of
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP ${.esxi.host_ip},${.esxi.message}\n")
When syslog-ng receives a syslog message, it logged it as;
=== system,error,critical, HOST IP ,
This means that $PROGRAM contains "system"
Now for the patterndb part.
The patterndb parser FIRST matches $PROGRAM To the <pattern>XXXX</pattern> in the <ruleset>
<?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>XXXX</pattern>
In your case you have specified <pattern>ESXI</pattern> so the patterndb parser will NOT use any of your patterndb because it does not match the $PROGRAM
You need to use
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>system</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
You have not included a complete syslong-ng source line for me to see what you are trying to match against so I can not tell if you pattern will actually match the lines that you are trying to match. At my organization we run ESX as well, and none of our lines would match the pattern that you have, but your environment might be different.
I hope this was more clear.
Evan.
________________________________________ From: 不坏阿峰 [onlydebian@gmail.com] Sent: Sunday, April 28, 2013 8:24 AM To: syslog-ng@lists.balabit.hu; Evan Rempel Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there.
how to do on this ----->>>> If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work.
2013/4/28 <syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>> Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>
To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>
You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu<mailto: syslog-ng-owner@lists.balabit.hu>
When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (????) 2. Re: Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (Evan Rempel)
----------------------------------------------------------------------
Message: 1 Date: Sat, 27 Apr 2013 22:34:50 +0800 From: ???? <onlydebian@gmail.com<mailto:onlydebian@gmail.com>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Message-ID: <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR= V7wJ1EJdvE6Zx9zg@mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com>> Content-Type: text/plain; charset="iso-8859-1"
when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time}
test log output ,just like this. === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP ,
do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot.
root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE" Pattern matching part: @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@ @ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ Matching part: Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE Values: MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE PROGRAM=ESXI .classifier.class=esxi .classifier.rule_id=182437592347598 .esxi.month=Apr .esxi.date=26 .esxi.time=15:17:31 .esxi.host_ip=192.168.88.71 .esxi.program= vmkernel .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE root@debian:~#
my configuration like as below
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>ESXI</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
######## syslog-ng.conf ########
#####Parser##### parser pattern_db { db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); };
#Check pattern matching destination udp_esxi_output { file("/var/log/pattern_output" template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n") template_escape(no)); };
#####Log##### log { source(s_network); parser(pattern_db); destination(udp_esxi_output); };
That looks more like what I would expect. In your example source line your $PROGRAM will be vmkernel and should be the text in the <pattern></pattern> xml tag. Aslo, your pattern needs to start at the text following the vmkernel: part of the syslog line. Only the $MESSAGE part of the syslog line is sent to the patterndb for parsing, unless your source definition in the syslog-ng.conf file has the flags(no-parse) option, but that would be unusual. Evan Evan Rempel 250.271.7691 University Systems, University of Victoria 不坏阿峰 <onlydebian@gmail.com> wrote: sorry for miss the purpose what i want to do. (1) first . receive syslog from esxi host from UDP . (done) (2) second. parse the log from UDP and parse with pattern db and get separate imformation ( meet the problem i ask for help) (3)third. store separate infor to Oracle table(done, test successfully on syslog-ng macro value) for the second step, i use the way file() to check the situation of db-parse. some sample log message from esxi host. Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP: mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0) Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default'] Unrecognized 2013/4/29 不坏阿峰 <onlydebian@gmail.com<mailto:onlydebian@gmail.com>> attachment is my current syslog-ng.conf. and esxi_pattern.xml. my syslog-ng receive UDP log from esxi host and try to test the db-parse and log it. i have change to <pattern>system</pattern>, but still can not get value from parse refer macro. thanks. 2013/4/28 Evan Rempel <erempel@uvic.ca<mailto:erempel@uvic.ca>> Sorry for not being more clear in my first response. You have a template of template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP ${.esxi.host_ip},${.esxi.message}\n") When syslog-ng receives a syslog message, it logged it as; === system,error,critical, HOST IP , This means that $PROGRAM contains "system" Now for the patterndb part. The patterndb parser FIRST matches $PROGRAM To the <pattern>XXXX</pattern> in the <ruleset> <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>XXXX</pattern> In your case you have specified <pattern>ESXI</pattern> so the patterndb parser will NOT use any of your patterndb because it does not match the $PROGRAM You need to use ######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>system</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> You have not included a complete syslong-ng source line for me to see what you are trying to match against so I can not tell if you pattern will actually match the lines that you are trying to match. At my organization we run ESX as well, and none of our lines would match the pattern that you have, but your environment might be different. I hope this was more clear. Evan. ________________________________________ From: 不坏阿峰 [onlydebian@gmail.com<mailto:onlydebian@gmail.com>] Sent: Sunday, April 28, 2013 8:24 AM To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>; Evan Rempel Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there. how to do on this ----->>>> If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work. 2013/4/28 <syslog-ng-request@lists.balabit.hu<mailto:syslog-ng-request@lists.balabit.hu><mailto:syslog-ng-request@lists.balabit.hu<mailto:syslog-ng-request@lists.balabit.hu>>> Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu<mailto:syslog-ng-request@lists.balabit.hu><mailto:syslog-ng-request@lists.balabit.hu<mailto:syslog-ng-request@lists.balabit.hu>> You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu<mailto:syslog-ng-owner@lists.balabit.hu><mailto:syslog-ng-owner@lists.balabit.hu<mailto:syslog-ng-owner@lists.balabit.hu>> When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..." Today's Topics: 1. Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (????) 2. Re: Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (Evan Rempel) ---------------------------------------------------------------------- Message: 1 Date: Sat, 27 Apr 2013 22:34:50 +0800 From: ???? <onlydebian@gmail.com<mailto:onlydebian@gmail.com><mailto:onlydebian@gmail.com<mailto:onlydebian@gmail.com>>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Message-ID: <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR=V7wJ1EJdvE6Zx9zg@mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com><mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com>>> Content-Type: text/plain; charset="iso-8859-1" when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time} test log output ,just like this. === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot. root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE" Pattern matching part: @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ Matching part: Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE Values: MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE PROGRAM=ESXI .classifier.class=esxi .classifier.rule_id=182437592347598 .esxi.month=Apr .esxi.date=26 .esxi.time=15:17:31 .esxi.host_ip=192.168.88.71 .esxi.program= vmkernel .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE root@debian:~# my configuration like as below ######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>ESXI</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> ######## syslog-ng.conf ######## #####Parser##### parser pattern_db { db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); }; #Check pattern matching destination udp_esxi_output { file("/var/log/pattern_output" template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n") template_escape(no)); }; #####Log##### log { source(s_network); parser(pattern_db); destination(udp_esxi_output); }; -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/20e80756/at... ------------------------------ Message: 2 Date: Sat, 27 Apr 2013 16:10:02 +0000 From: Evan Rempel <erempel@uvic.ca<mailto:erempel@uvic.ca><mailto:erempel@uvic.ca<mailto:erempel@uvic.ca>>> Subject: Re: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) To: "Syslog-ng users' and developers' mailing list" <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>>> Message-ID: <q8vb966l4qe0219lsusm5ju4.1367078999747@email.android.com<mailto:q8vb966l4qe0219lsusm5ju4.1367078999747@email.android.com><mailto:q8vb966l4qe0219lsusm5ju4.1367078999747@email.android.com<mailto:q8vb966l4qe0219lsusm5ju4.1367078999747@email.android.com>>> Content-Type: text/plain; charset="iso-2022-jp" It would appear that you have everything correct when the "PROGRAM" is ESXI but the log line as syslog-ng sees it has a PROGRAM of "system" according to your test log output. If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work. Evan Rempel 250.271.7691<tel:250.271.7691><tel:250.271.7691<tel:250.271.7691>> University Systems, University of Victoria ???? <onlydebian@gmail.com<mailto:onlydebian@gmail.com><mailto:onlydebian@gmail.com<mailto:onlydebian@gmail.com>>> wrote: when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time} test log output ,just like this. === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot. root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE" Pattern matching part: @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@ @IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ @ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ Matching part: Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE Values: MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE PROGRAM=ESXI .classifier.class=esxi .classifier.rule_id=182437592347598 .esxi.month=Apr .esxi.date=26 .esxi.time=15:17:31 .esxi.host_ip=192.168.88.71 .esxi.program= vmkernel .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE root@debian:~# my configuration like as below ######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>ESXI</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@ @IPv4:.esxi.host_ip:@@ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb> ######## syslog-ng.conf ######## #####Parser##### parser pattern_db { db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); }; #Check pattern matching destination udp_esxi_output { file("/var/log/pattern_output" template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n") template_escape(no)); }; #####Log##### log { source(s_network); parser(pattern_db); destination(udp_esxi_output); }; -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/2f67c039/at... ------------------------------ _______________________________________________ syslog-ng maillist - syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu><mailto:syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> https://lists.balabit.hu/mailman/listinfo/syslog-ng End of syslog-ng Digest, Vol 96, Issue 25 *****************************************
i have try put <pattern>vmkernel</pattern> <pattern>hostd-probe</pattern> do test, not work. 2013/4/29 Evan Rempel <erempel@uvic.ca>
That looks more like what I would expect. In your example source line your $PROGRAM will be vmkernel and should be the text in the <pattern></pattern> xml tag.
Aslo, your pattern needs to start at the text following the vmkernel: part of the syslog line. Only the $MESSAGE part of the syslog line is sent to the patterndb for parsing, unless your source definition in the syslog-ng.conf file has the flags(no-parse) option, but that would be unusual.
Evan
Evan Rempel 250.271.7691 University Systems, University of Victoria
不坏阿峰 <onlydebian@gmail.com> wrote:
sorry for miss the purpose what i want to do. (1) first . receive syslog from esxi host from UDP . (done) (2) second. parse the log from UDP and parse with pattern db and get separate imformation ( meet the problem i ask for help) (3)third. store separate infor to Oracle table(done, test successfully on syslog-ng macro value)
for the second step, i use the way file() to check the situation of db-parse.
some sample log message from esxi host.
Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP: mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0) Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default'] Unrecognized
2013/4/29 不坏阿峰 <onlydebian@gmail.com>
attachment is my current syslog-ng.conf. and esxi_pattern.xml.
my syslog-ng receive UDP log from esxi host and try to test the db-parse and log it.
i have change to <pattern>system</pattern>, but still can not get value from parse refer macro.
thanks.
2013/4/28 Evan Rempel <erempel@uvic.ca>
Sorry for not being more clear in my first response.
You have a template of
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP ${.esxi.host_ip},${.esxi.message}\n")
When syslog-ng receives a syslog message, it logged it as;
=== system,error,critical, HOST IP ,
This means that $PROGRAM contains "system"
Now for the patterndb part.
The patterndb parser FIRST matches $PROGRAM To the <pattern>XXXX</pattern> in the <ruleset>
<?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>XXXX</pattern>
In your case you have specified <pattern>ESXI</pattern> so the patterndb parser will NOT use any of your patterndb because it does not match the $PROGRAM
You need to use
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>system</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
You have not included a complete syslong-ng source line for me to see what you are trying to match against so I can not tell if you pattern will actually match the lines that you are trying to match. At my organization we run ESX as well, and none of our lines would match the pattern that you have, but your environment might be different.
I hope this was more clear.
Evan.
________________________________________ From: 不坏阿峰 [onlydebian@gmail.com] Sent: Sunday, April 28, 2013 8:24 AM To: syslog-ng@lists.balabit.hu; Evan Rempel Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there.
how to do on this ----->>>> If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work.
2013/4/28 <syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>> Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>
To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>
You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu<mailto: syslog-ng-owner@lists.balabit.hu>
When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (????) 2. Re: Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (Evan Rempel)
----------------------------------------------------------------------
Message: 1 Date: Sat, 27 Apr 2013 22:34:50 +0800 From: ???? <onlydebian@gmail.com<mailto:onlydebian@gmail.com>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Message-ID: <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR= V7wJ1EJdvE6Zx9zg@mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com>> Content-Type: text/plain; charset="iso-8859-1"
when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time}
test log output ,just like this. === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP ,
do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot.
root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE" Pattern matching part: @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@ @ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ Matching part: Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE Values: MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE PROGRAM=ESXI .classifier.class=esxi .classifier.rule_id=182437592347598 .esxi.month=Apr .esxi.date=26 .esxi.time=15:17:31 .esxi.host_ip=192.168.88.71 .esxi.program= vmkernel .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE root@debian:~#
my configuration like as below
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>ESXI</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
######## syslog-ng.conf ########
#####Parser##### parser pattern_db { db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); };
#Check pattern matching destination udp_esxi_output { file("/var/log/pattern_output" template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n") template_escape(no)); };
#####Log##### log { source(s_network); parser(pattern_db); destination(udp_esxi_output); };
Dear Marton, Martin i saw your post in mail list. could you give me some advice to solve my problem. why i can not get db-parse macro value from syslog-ng while pdbtool match work successfully. thanks. my thread is here https://lists.balabit.hu/pipermail/syslog-ng/2013-April/020300.html 2013/4/29 不坏阿峰 <onlydebian@gmail.com>
i have try put <pattern>vmkernel</pattern> <pattern>hostd-probe</pattern> do test, not work.
2013/4/29 Evan Rempel <erempel@uvic.ca>
That looks more like what I would expect. In your example source line your $PROGRAM will be vmkernel and should be the text in the <pattern></pattern> xml tag.
Aslo, your pattern needs to start at the text following the vmkernel: part of the syslog line. Only the $MESSAGE part of the syslog line is sent to the patterndb for parsing, unless your source definition in the syslog-ng.conf file has the flags(no-parse) option, but that would be unusual.
Evan
Evan Rempel 250.271.7691 University Systems, University of Victoria
不坏阿峰 <onlydebian@gmail.com> wrote:
sorry for miss the purpose what i want to do. (1) first . receive syslog from esxi host from UDP . (done) (2) second. parse the log from UDP and parse with pattern db and get separate imformation ( meet the problem i ask for help) (3)third. store separate infor to Oracle table(done, test successfully on syslog-ng macro value)
for the second step, i use the way file() to check the situation of db-parse.
some sample log message from esxi host.
Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP: mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0) Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default'] Unrecognized
2013/4/29 不坏阿峰 <onlydebian@gmail.com>
attachment is my current syslog-ng.conf. and esxi_pattern.xml.
my syslog-ng receive UDP log from esxi host and try to test the db-parse and log it.
i have change to <pattern>system</pattern>, but still can not get value from parse refer macro.
thanks.
2013/4/28 Evan Rempel <erempel@uvic.ca>
Sorry for not being more clear in my first response.
You have a template of
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP ${.esxi.host_ip},${.esxi.message}\n")
When syslog-ng receives a syslog message, it logged it as;
=== system,error,critical, HOST IP ,
This means that $PROGRAM contains "system"
Now for the patterndb part.
The patterndb parser FIRST matches $PROGRAM To the <pattern>XXXX</pattern> in the <ruleset>
<?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>XXXX</pattern>
In your case you have specified <pattern>ESXI</pattern> so the patterndb parser will NOT use any of your patterndb because it does not match the $PROGRAM
You need to use
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>system</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
You have not included a complete syslong-ng source line for me to see what you are trying to match against so I can not tell if you pattern will actually match the lines that you are trying to match. At my organization we run ESX as well, and none of our lines would match the pattern that you have, but your environment might be different.
I hope this was more clear.
Evan.
________________________________________ From: 不坏阿峰 [onlydebian@gmail.com] Sent: Sunday, April 28, 2013 8:24 AM To: syslog-ng@lists.balabit.hu; Evan Rempel Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there.
how to do on this ----->>>> If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work.
2013/4/28 <syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>> Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>
To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>
You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu<mailto: syslog-ng-owner@lists.balabit.hu>
When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (????) 2. Re: Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (Evan Rempel)
----------------------------------------------------------------------
Message: 1 Date: Sat, 27 Apr 2013 22:34:50 +0800 From: ???? <onlydebian@gmail.com<mailto:onlydebian@gmail.com>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Message-ID: <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR= V7wJ1EJdvE6Zx9zg@mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com
Content-Type: text/plain; charset="iso-8859-1"
when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time}
test log output ,just like this. === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP ,
do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot.
root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE" Pattern matching part: @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@ @ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ Matching part: Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE Values: MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE PROGRAM=ESXI .classifier.class=esxi .classifier.rule_id=182437592347598 .esxi.month=Apr .esxi.date=26 .esxi.time=15:17:31 .esxi.host_ip=192.168.88.71 .esxi.program= vmkernel .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE root@debian:~#
my configuration like as below
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>ESXI</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
######## syslog-ng.conf ########
#####Parser##### parser pattern_db { db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); };
#Check pattern matching destination udp_esxi_output { file("/var/log/pattern_output" template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n") template_escape(no)); };
#####Log##### log { source(s_network); parser(pattern_db); destination(udp_esxi_output); };
The issue is that your test message is actually the raw message as seen in a log file containing the timestamp, host, and program as well as the message. In the syslog-ng.conf, that would correspond to $S_TIMESTAMP, $HOST, $PROGRAM, and $MSG_ONLY. Your pattern isn't actually extracting any fields not already extracted by syslog-ng without the patterndb. Are you collecting this file using the file() driver or are you reading from the network? If this is a raw file you are reading, then there is no program, and so you just need to remove the first <pattern> element entirely to say to match on all programs, or use the program_override() flag to set a program explicitly and make sure it matches the program in the <pattern> element. On Mon, Apr 29, 2013 at 4:35 AM, 不坏阿峰 <onlydebian@gmail.com> wrote:
Dear Marton, Martin
i saw your post in mail list. could you give me some advice to solve my problem. why i can not get db-parse macro value from syslog-ng while pdbtool match work successfully.
thanks.
my thread is here
https://lists.balabit.hu/pipermail/syslog-ng/2013-April/020300.html
2013/4/29 不坏阿峰 <onlydebian@gmail.com>
i have try put <pattern>vmkernel</pattern> <pattern>hostd-probe</pattern> do test, not work.
2013/4/29 Evan Rempel <erempel@uvic.ca>
That looks more like what I would expect. In your example source line your $PROGRAM will be vmkernel and should be the text in the <pattern></pattern> xml tag.
Aslo, your pattern needs to start at the text following the vmkernel: part of the syslog line. Only the $MESSAGE part of the syslog line is sent to the patterndb for parsing, unless your source definition in the syslog-ng.conf file has the flags(no-parse) option, but that would be unusual.
Evan
Evan Rempel 250.271.7691 University Systems, University of Victoria
不坏阿峰 <onlydebian@gmail.com> wrote:
sorry for miss the purpose what i want to do. (1) first . receive syslog from esxi host from UDP . (done) (2) second. parse the log from UDP and parse with pattern db and get separate imformation ( meet the problem i ask for help) (3)third. store separate infor to Oracle table(done, test successfully on syslog-ng macro value)
for the second step, i use the way file() to check the situation of db-parse.
some sample log message from esxi host.
Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP: mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0) Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default'] Unrecognized
2013/4/29 不坏阿峰 <onlydebian@gmail.com>
attachment is my current syslog-ng.conf. and esxi_pattern.xml.
my syslog-ng receive UDP log from esxi host and try to test the db-parse and log it.
i have change to <pattern>system</pattern>, but still can not get value from parse refer macro.
thanks.
2013/4/28 Evan Rempel <erempel@uvic.ca>
Sorry for not being more clear in my first response.
You have a template of
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP ${.esxi.host_ip},${.esxi.message}\n")
When syslog-ng receives a syslog message, it logged it as;
=== system,error,critical, HOST IP ,
This means that $PROGRAM contains "system"
Now for the patterndb part.
The patterndb parser FIRST matches $PROGRAM To the <pattern>XXXX</pattern> in the <ruleset>
<?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>XXXX</pattern>
In your case you have specified <pattern>ESXI</pattern> so the patterndb parser will NOT use any of your patterndb because it does not match the $PROGRAM
You need to use
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>system</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
You have not included a complete syslong-ng source line for me to see what you are trying to match against so I can not tell if you pattern will actually match the lines that you are trying to match. At my organization we run ESX as well, and none of our lines would match the pattern that you have, but your environment might be different.
I hope this was more clear.
Evan.
________________________________________ From: 不坏阿峰 [onlydebian@gmail.com] Sent: Sunday, April 28, 2013 8:24 AM To: syslog-ng@lists.balabit.hu; Evan Rempel Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there.
how to do on this ----->>>> If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work.
2013/4/28 <syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>> Send syslog-ng mailing list submissions to syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>
To subscribe or unsubscribe via the World Wide Web, visit https://lists.balabit.hu/mailman/listinfo/syslog-ng or, via email, send a message with subject or body 'help' to syslog-ng-request@lists.balabit.hu<mailto: syslog-ng-request@lists.balabit.hu>
You can reach the person managing the list at syslog-ng-owner@lists.balabit.hu<mailto: syslog-ng-owner@lists.balabit.hu>
When replying, please edit your Subject line so it is more specific than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (????) 2. Re: Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) (Evan Rempel)
----------------------------------------------------------------------
Message: 1 Date: Sat, 27 Apr 2013 22:34:50 +0800 From: ???? <onlydebian@gmail.com<mailto:onlydebian@gmail.com>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze) To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> Message-ID: <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR= V7wJ1EJdvE6Zx9zg@mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com
> Content-Type: text/plain; charset="iso-8859-1"
when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time}
test log output ,just like this. === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP , === system,error,critical, HOST IP ,
do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot.
root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE" Pattern matching part: @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@ @ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ Matching part: Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE Values: MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE PROGRAM=ESXI .classifier.class=esxi .classifier.rule_id=182437592347598 .esxi.month=Apr .esxi.date=26 .esxi.time=15:17:31 .esxi.host_ip=192.168.88.71 .esxi.program= vmkernel .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE root@debian:~#
my configuration like as below
######## esxi_pattern.xml ############ <?xml version="1.0" encoding="utf-8"?> <patterndb version='3' pub_date='2009-04-17'> <ruleset name='esxi' id='123456678'> <pattern>ESXI</pattern> <rules> <rule provider='Fone Bro' id='182437592347598' class='esxi'> <patterns> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@ @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern> </patterns> </rule> </rules> </ruleset> </patterndb>
######## syslog-ng.conf ########
#####Parser##### parser pattern_db { db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml")); };
#Check pattern matching destination udp_esxi_output { file("/var/log/pattern_output" template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n") template_escape(no)); };
#####Log##### log { source(s_network); parser(pattern_db); destination(udp_esxi_output); };
participants (3)
-
Evan Rempel
-
Martin Holste
-
不坏阿峰