[syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
不坏阿峰
onlydebian at gmail.com
Sun Apr 28 18:23:31 CEST 2013
sorry for miss the purpose what i want to do.
(1) first . receive syslog from esxi host from UDP . (done)
(2) second. parse the log from UDP and parse with pattern db and get
separate imformation ( meet the problem i ask for help)
(3)third. store separate infor to Oracle table(done, test successfully on
syslog-ng macro value)
for the second step, i use the way file() to check the situation of
db-parse.
some sample log message from esxi host.
Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP:
mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0)
Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default']
Unrecognized
2013/4/29 不坏阿峰 <onlydebian at gmail.com>
> attachment is my current syslog-ng.conf. and esxi_pattern.xml.
>
> my syslog-ng receive UDP log from esxi host and try to test the db-parse
> and log it.
>
> i have change to <pattern>system</pattern>, but still can not get value
> from parse refer macro.
>
> thanks.
>
>
> 2013/4/28 Evan Rempel <erempel at uvic.ca>
>
>> Sorry for not being more clear in my first response.
>>
>> You have a template of
>>
>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP
>> ${.esxi.host_ip},${.esxi.message}\n")
>>
>> When syslog-ng receives a syslog message, it logged it as;
>>
>> === system,error,critical, HOST IP ,
>>
>> This means that $PROGRAM contains "system"
>>
>> Now for the patterndb part.
>>
>> The patterndb parser FIRST matches $PROGRAM To the
>> <pattern>XXXX</pattern> in the <ruleset>
>>
>> <?xml version="1.0" encoding="utf-8"?>
>> <patterndb version='3' pub_date='2009-04-17'>
>> <ruleset name='esxi' id='123456678'>
>> <pattern>XXXX</pattern>
>>
>> In your case you have specified <pattern>ESXI</pattern> so the patterndb
>> parser will NOT use any
>> of your patterndb because it does not match the $PROGRAM
>>
>> You need to use
>>
>> ######## esxi_pattern.xml ############
>> <?xml version="1.0" encoding="utf-8"?>
>> <patterndb version='3' pub_date='2009-04-17'>
>> <ruleset name='esxi' id='123456678'>
>> <pattern>system</pattern>
>> <rules>
>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>> <patterns>
>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
>> @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>> </patterns>
>> </rule>
>> </rules>
>> </ruleset>
>> </patterndb>
>>
>>
>>
>> You have not included a complete syslong-ng source line for me to see
>> what you are trying to match against so I can
>> not tell if you pattern will actually match the lines that you are trying
>> to match.
>> At my organization we run ESX as well, and none of our lines would match
>> the pattern that you have, but
>> your environment might be different.
>>
>> I hope this was more clear.
>>
>> Evan.
>>
>>
>>
>>
>> ________________________________________
>> From: 不坏阿峰 [onlydebian at gmail.com]
>> Sent: Sunday, April 28, 2013 8:24 AM
>> To: syslog-ng at lists.balabit.hu; Evan Rempel
>> Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian
>> squeeze)
>>
>> thanks to your reply. i do not understand how to do now. it puzzle and
>> trouble me some days. i read the balabit syslog-ng OSE guide documents
>> and only have simple information in there.
>>
>> how to do on this
>> ----->>>>
>> If you change the patterndb ruleset pattern to use a program of system
>> rather than ESXI I think it would work.
>>
>>
>> 2013/4/28 <syslog-ng-request at lists.balabit.hu<mailto:
>> syslog-ng-request at lists.balabit.hu>>
>> Send syslog-ng mailing list submissions to
>> syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> or, via email, send a message with subject or body 'help' to
>> syslog-ng-request at lists.balabit.hu<mailto:
>> syslog-ng-request at lists.balabit.hu>
>>
>> You can reach the person managing the list at
>> syslog-ng-owner at lists.balabit.hu<mailto:
>> syslog-ng-owner at lists.balabit.hu>
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of syslog-ng digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Can not get DBParse match macro result (syslog-ng 3.13
>> debian squeeze) (????)
>> 2. Re: Can not get DBParse match macro result (syslog-ng 3.13
>> debian squeeze) (Evan Rempel)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Sat, 27 Apr 2013 22:34:50 +0800
>> From: ???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com>>
>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng
>> 3.13 debian squeeze)
>> To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
>> Message-ID:
>> <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR=
>> V7wJ1EJdvE6Zx9zg at mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg at mail.gmail.com>>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> when use pdbtool do match test, it is success. but from syslog-ng can not
>> return result of macro
>> i can not get macro result. for example, ${.esxi.month} no value, same
>> as ${.esxi.host_ip} ${.esxi.time}
>>
>> test log output ,just like this.
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>>
>>
>> do the pdbtool test, it's ok. wish someone can give me some solution and
>> help. i have search some mail list but i can not get the right solution.
>> thanks a lot.
>>
>> root at debian:~# pdbtool match -D -c -p
>> /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31
>> 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
>> 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path
>> "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20
>> 0x0.
>> Act:NONE"
>> Pattern matching part:
>> @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@
>> @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@
>> @ESTRING:.esxi.program=
>> vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
>> Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP:
>> nmp_ThrottleLogForDevice:2319: Cmd
>> 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path
>> vmhba0:C0:T0:L0
>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
>> Matching part:
>> Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0
>> Valid
>> sense data: 0x5 0x20 0x0. Act:NONE
>> Values:
>> MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0
>> Valid
>> sense data: 0x5 0x20 0x0. Act:NONE
>> PROGRAM=ESXI
>> .classifier.class=esxi
>> .classifier.rule_id=182437592347598
>> .esxi.month=Apr
>> .esxi.date=26
>> .esxi.time=15:17:31
>> .esxi.host_ip=192.168.88.71
>> .esxi.program= vmkernel
>> .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
>> root at debian:~#
>>
>>
>> my configuration like as below
>>
>> ######## esxi_pattern.xml ############
>> <?xml version="1.0" encoding="utf-8"?>
>> <patterndb version='3' pub_date='2009-04-17'>
>> <ruleset name='esxi' id='123456678'>
>> <pattern>ESXI</pattern>
>> <rules>
>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>> <patterns>
>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
>> @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>> </patterns>
>> </rule>
>> </rules>
>> </ruleset>
>> </patterndb>
>>
>> ######## syslog-ng.conf ########
>>
>> #####Parser#####
>> parser pattern_db {
>> db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
>> };
>>
>> #Check pattern matching
>> destination udp_esxi_output {
>> file("/var/log/pattern_output"
>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST
>> IP ${.esxi.host_ip},${.esxi.message}\n")
>> template_escape(no));
>> };
>>
>> #####Log#####
>> log {
>> source(s_network);
>> parser(pattern_db);
>> destination(udp_esxi_output);
>> };
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/20e80756/attachment.html
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Sat, 27 Apr 2013 16:10:02 +0000
>> From: Evan Rempel <erempel at uvic.ca<mailto:erempel at uvic.ca>>
>> Subject: Re: [syslog-ng] Can not get DBParse match macro result
>> (syslog-ng 3.13 debian squeeze)
>> To: "Syslog-ng users' and developers' mailing list"
>> <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
>> Message-ID: <q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com
>> <mailto:q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com>>
>> Content-Type: text/plain; charset="iso-2022-jp"
>>
>> It would appear that you have everything correct when the "PROGRAM" is
>> ESXI but the log line as syslog-ng sees it has a PROGRAM of "system"
>> according to your test log output.
>>
>> If you change the patterndb ruleset pattern to use a program of system
>> rather than ESXI I think it would work.
>>
>>
>> Evan Rempel 250.271.7691<tel:250.271.7691>
>> University Systems, University of Victoria
>>
>> ???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com>> wrote:
>>
>>
>>
>> when use pdbtool do match test, it is success. but from syslog-ng can not
>> return result of macro
>> i can not get macro result. for example, ${.esxi.month} no value,
>> same as ${.esxi.host_ip} ${.esxi.time}
>>
>> test log output ,just like this.
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>> === system,error,critical, HOST IP ,
>>
>>
>> do the pdbtool test, it's ok. wish someone can give me some solution and
>> help. i have search some mail list but i can not get the right solution.
>> thanks a lot.
>>
>> root at debian:~# pdbtool match -D -c -p
>> /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31
>> 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
>> 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path
>> "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
>> Act:NONE"
>> Pattern matching part:
>> @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@@STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program=
>> vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
>> 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
>> Matching part:
>> Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
>> sense data: 0x5 0x20 0x0. Act:NONE
>> Values:
>> MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
>> sense data: 0x5 0x20 0x0. Act:NONE
>> PROGRAM=ESXI
>> .classifier.class=esxi
>> .classifier.rule_id=182437592347598
>> .esxi.month=Apr
>> .esxi.date=26
>> .esxi.time=15:17:31
>> .esxi.host_ip=192.168.88.71
>> .esxi.program= vmkernel
>> .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
>> root at debian:~#
>>
>>
>> my configuration like as below
>>
>> ######## esxi_pattern.xml ############
>> <?xml version="1.0" encoding="utf-8"?>
>> <patterndb version='3' pub_date='2009-04-17'>
>> <ruleset name='esxi' id='123456678'>
>> <pattern>ESXI</pattern>
>> <rules>
>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>> <patterns>
>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>> </patterns>
>> </rule>
>> </rules>
>> </ruleset>
>> </patterndb>
>>
>> ######## syslog-ng.conf ########
>>
>> #####Parser#####
>> parser pattern_db {
>> db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
>> };
>>
>> #Check pattern matching
>> destination udp_esxi_output {
>> file("/var/log/pattern_output"
>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST
>> IP ${.esxi.host_ip},${.esxi.message}\n")
>> template_escape(no));
>> };
>>
>> #####Log#####
>> log {
>> source(s_network);
>> parser(pattern_db);
>> destination(udp_esxi_output);
>> };
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL:
>> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/2f67c039/attachment-0001.htm
>>
>> ------------------------------
>>
>> _______________________________________________
>> syslog-ng maillist - syslog-ng at lists.balabit.hu<mailto:
>> syslog-ng at lists.balabit.hu>
>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>
>>
>> End of syslog-ng Digest, Vol 96, Issue 25
>> *****************************************
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130429/be9edfd2/attachment-0001.htm
More information about the syslog-ng
mailing list