[syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
不坏阿峰
onlydebian at gmail.com
Mon Apr 29 06:25:31 CEST 2013
i have try put <pattern>vmkernel</pattern> <pattern>hostd-probe</pattern> do
test, not work.
2013/4/29 Evan Rempel <erempel at uvic.ca>
> That looks more like what I would expect.
> In your example source line your $PROGRAM will be vmkernel and should be the text in the <pattern></pattern> xml tag.
>
> Aslo, your pattern needs to start at the text following the vmkernel: part of the syslog line. Only the $MESSAGE part of the syslog line is sent to the patterndb for parsing, unless your source definition in the syslog-ng.conf file has the flags(no-parse) option, but that would be unusual.
>
> Evan
>
>
> Evan Rempel 250.271.7691
> University Systems, University of Victoria
>
> 不坏阿峰 <onlydebian at gmail.com> wrote:
>
>
> sorry for miss the purpose what i want to do.
> (1) first . receive syslog from esxi host from UDP . (done)
> (2) second. parse the log from UDP and parse with pattern db and get
> separate imformation ( meet the problem i ask for help)
> (3)third. store separate infor to Oracle table(done, test successfully on
> syslog-ng macro value)
>
> for the second step, i use the way file() to check the situation of
> db-parse.
>
> some sample log message from esxi host.
>
> Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP:
> mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0)
> Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default']
> Unrecognized
>
>
>
>
>
> 2013/4/29 不坏阿峰 <onlydebian at gmail.com>
>
>> attachment is my current syslog-ng.conf. and esxi_pattern.xml.
>>
>> my syslog-ng receive UDP log from esxi host and try to test the
>> db-parse and log it.
>>
>> i have change to <pattern>system</pattern>, but still can not get value
>> from parse refer macro.
>>
>> thanks.
>>
>>
>> 2013/4/28 Evan Rempel <erempel at uvic.ca>
>>
>>> Sorry for not being more clear in my first response.
>>>
>>> You have a template of
>>>
>>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP
>>> ${.esxi.host_ip},${.esxi.message}\n")
>>>
>>> When syslog-ng receives a syslog message, it logged it as;
>>>
>>> === system,error,critical, HOST IP ,
>>>
>>> This means that $PROGRAM contains "system"
>>>
>>> Now for the patterndb part.
>>>
>>> The patterndb parser FIRST matches $PROGRAM To the
>>> <pattern>XXXX</pattern> in the <ruleset>
>>>
>>> <?xml version="1.0" encoding="utf-8"?>
>>> <patterndb version='3' pub_date='2009-04-17'>
>>> <ruleset name='esxi' id='123456678'>
>>> <pattern>XXXX</pattern>
>>>
>>> In your case you have specified <pattern>ESXI</pattern> so the patterndb
>>> parser will NOT use any
>>> of your patterndb because it does not match the $PROGRAM
>>>
>>> You need to use
>>>
>>> ######## esxi_pattern.xml ############
>>> <?xml version="1.0" encoding="utf-8"?>
>>> <patterndb version='3' pub_date='2009-04-17'>
>>> <ruleset name='esxi' id='123456678'>
>>> <pattern>system</pattern>
>>> <rules>
>>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>>> <patterns>
>>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
>>> @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>>> </patterns>
>>> </rule>
>>> </rules>
>>> </ruleset>
>>> </patterndb>
>>>
>>>
>>>
>>> You have not included a complete syslong-ng source line for me to see
>>> what you are trying to match against so I can
>>> not tell if you pattern will actually match the lines that you are
>>> trying to match.
>>> At my organization we run ESX as well, and none of our lines would match
>>> the pattern that you have, but
>>> your environment might be different.
>>>
>>> I hope this was more clear.
>>>
>>> Evan.
>>>
>>>
>>>
>>>
>>> ________________________________________
>>> From: 不坏阿峰 [onlydebian at gmail.com]
>>> Sent: Sunday, April 28, 2013 8:24 AM
>>> To: syslog-ng at lists.balabit.hu; Evan Rempel
>>> Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13
>>> debian squeeze)
>>>
>>> thanks to your reply. i do not understand how to do now. it puzzle and
>>> trouble me some days. i read the balabit syslog-ng OSE guide documents
>>> and only have simple information in there.
>>>
>>> how to do on this
>>> ----->>>>
>>> If you change the patterndb ruleset pattern to use a program of system
>>> rather than ESXI I think it would work.
>>>
>>>
>>> 2013/4/28 <syslog-ng-request at lists.balabit.hu<mailto:
>>> syslog-ng-request at lists.balabit.hu>>
>>> Send syslog-ng mailing list submissions to
>>> syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>> or, via email, send a message with subject or body 'help' to
>>> syslog-ng-request at lists.balabit.hu<mailto:
>>> syslog-ng-request at lists.balabit.hu>
>>>
>>> You can reach the person managing the list at
>>> syslog-ng-owner at lists.balabit.hu<mailto:
>>> syslog-ng-owner at lists.balabit.hu>
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of syslog-ng digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>> 1. Can not get DBParse match macro result (syslog-ng 3.13
>>> debian squeeze) (????)
>>> 2. Re: Can not get DBParse match macro result (syslog-ng 3.13
>>> debian squeeze) (Evan Rempel)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Sat, 27 Apr 2013 22:34:50 +0800
>>> From: ???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com>>
>>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng
>>> 3.13 debian squeeze)
>>> To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
>>> Message-ID:
>>> <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR=
>>> V7wJ1EJdvE6Zx9zg at mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg at mail.gmail.com>>
>>> Content-Type: text/plain; charset="iso-8859-1"
>>>
>>> when use pdbtool do match test, it is success. but from syslog-ng can not
>>> return result of macro
>>> i can not get macro result. for example, ${.esxi.month} no value,
>>> same
>>> as ${.esxi.host_ip} ${.esxi.time}
>>>
>>> test log output ,just like this.
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>>
>>>
>>> do the pdbtool test, it's ok. wish someone can give me some solution and
>>> help. i have search some mail list but i can not get the right solution.
>>> thanks a lot.
>>>
>>> root at debian:~# pdbtool match -D -c -p
>>> /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31
>>> 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319:
>>> Cmd
>>> 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path
>>> "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20
>>> 0x0.
>>> Act:NONE"
>>> Pattern matching part:
>>> @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@
>>> @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@
>>> @ESTRING:.esxi.program=
>>> vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
>>> Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP:
>>> nmp_ThrottleLogForDevice:2319: Cmd
>>> 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path
>>> vmhba0:C0:T0:L0
>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
>>> Matching part:
>>> Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0
>>> Valid
>>> sense data: 0x5 0x20 0x0. Act:NONE
>>> Values:
>>> MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0
>>> Valid
>>> sense data: 0x5 0x20 0x0. Act:NONE
>>> PROGRAM=ESXI
>>> .classifier.class=esxi
>>> .classifier.rule_id=182437592347598
>>> .esxi.month=Apr
>>> .esxi.date=26
>>> .esxi.time=15:17:31
>>> .esxi.host_ip=192.168.88.71
>>> .esxi.program= vmkernel
>>> .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
>>> root at debian:~#
>>>
>>>
>>> my configuration like as below
>>>
>>> ######## esxi_pattern.xml ############
>>> <?xml version="1.0" encoding="utf-8"?>
>>> <patterndb version='3' pub_date='2009-04-17'>
>>> <ruleset name='esxi' id='123456678'>
>>> <pattern>ESXI</pattern>
>>> <rules>
>>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>>> <patterns>
>>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
>>> @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>>> </patterns>
>>> </rule>
>>> </rules>
>>> </ruleset>
>>> </patterndb>
>>>
>>> ######## syslog-ng.conf ########
>>>
>>> #####Parser#####
>>> parser pattern_db {
>>> db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
>>> };
>>>
>>> #Check pattern matching
>>> destination udp_esxi_output {
>>> file("/var/log/pattern_output"
>>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST
>>> IP ${.esxi.host_ip},${.esxi.message}\n")
>>> template_escape(no));
>>> };
>>>
>>> #####Log#####
>>> log {
>>> source(s_network);
>>> parser(pattern_db);
>>> destination(udp_esxi_output);
>>> };
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL:
>>> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/20e80756/attachment.html
>>>
>>> ------------------------------
>>>
>>> Message: 2
>>> Date: Sat, 27 Apr 2013 16:10:02 +0000
>>> From: Evan Rempel <erempel at uvic.ca<mailto:erempel at uvic.ca>>
>>> Subject: Re: [syslog-ng] Can not get DBParse match macro result
>>> (syslog-ng 3.13 debian squeeze)
>>> To: "Syslog-ng users' and developers' mailing list"
>>> <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
>>> Message-ID: <q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com
>>> <mailto:q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com>>
>>> Content-Type: text/plain; charset="iso-2022-jp"
>>>
>>> It would appear that you have everything correct when the "PROGRAM" is
>>> ESXI but the log line as syslog-ng sees it has a PROGRAM of "system"
>>> according to your test log output.
>>>
>>> If you change the patterndb ruleset pattern to use a program of system
>>> rather than ESXI I think it would work.
>>>
>>>
>>> Evan Rempel 250.271.7691<tel:250.271.7691>
>>> University Systems, University of Victoria
>>>
>>> ???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com>> wrote:
>>>
>>>
>>>
>>> when use pdbtool do match test, it is success. but from syslog-ng can
>>> not return result of macro
>>> i can not get macro result. for example, ${.esxi.month} no value,
>>> same as ${.esxi.host_ip} ${.esxi.time}
>>>
>>> test log output ,just like this.
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>> === system,error,critical, HOST IP ,
>>>
>>>
>>> do the pdbtool test, it's ok. wish someone can give me some solution
>>> and help. i have search some mail list but i can not get the right
>>> solution. thanks a lot.
>>>
>>> root at debian:~# pdbtool match -D -c -p
>>> /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31
>>> 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
>>> 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path
>>> "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
>>> Act:NONE"
>>> Pattern matching part:
>>> @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@@STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program=
>>> vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
>>> 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
>>> Matching part:
>>> Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
>>> sense data: 0x5 0x20 0x0. Act:NONE
>>> Values:
>>> MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
>>> sense data: 0x5 0x20 0x0. Act:NONE
>>> PROGRAM=ESXI
>>> .classifier.class=esxi
>>> .classifier.rule_id=182437592347598
>>> .esxi.month=Apr
>>> .esxi.date=26
>>> .esxi.time=15:17:31
>>> .esxi.host_ip=192.168.88.71
>>> .esxi.program= vmkernel
>>> .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
>>> root at debian:~#
>>>
>>>
>>> my configuration like as below
>>>
>>> ######## esxi_pattern.xml ############
>>> <?xml version="1.0" encoding="utf-8"?>
>>> <patterndb version='3' pub_date='2009-04-17'>
>>> <ruleset name='esxi' id='123456678'>
>>> <pattern>ESXI</pattern>
>>> <rules>
>>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>>> <patterns>
>>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@@STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>>> </patterns>
>>> </rule>
>>> </rules>
>>> </ruleset>
>>> </patterndb>
>>>
>>> ######## syslog-ng.conf ########
>>>
>>> #####Parser#####
>>> parser pattern_db {
>>> db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
>>> };
>>>
>>> #Check pattern matching
>>> destination udp_esxi_output {
>>> file("/var/log/pattern_output"
>>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time}
>>> HOST IP ${.esxi.host_ip},${.esxi.message}\n")
>>> template_escape(no));
>>> };
>>>
>>> #####Log#####
>>> log {
>>> source(s_network);
>>> parser(pattern_db);
>>> destination(udp_esxi_output);
>>> };
>>>
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL:
>>> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/2f67c039/attachment-0001.htm
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> syslog-ng maillist - syslog-ng at lists.balabit.hu<mailto:
>>> syslog-ng at lists.balabit.hu>
>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>
>>>
>>> End of syslog-ng Digest, Vol 96, Issue 25
>>> *****************************************
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130429/164150a6/attachment-0001.htm
More information about the syslog-ng
mailing list