[syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
不坏阿峰
onlydebian at gmail.com
Mon Apr 29 11:35:35 CEST 2013
Dear Marton, Martin
i saw your post in mail list. could you give me some advice to solve my
problem. why i can not get db-parse macro value from syslog-ng while
pdbtool match work successfully.
thanks.
my thread is here
https://lists.balabit.hu/pipermail/syslog-ng/2013-April/020300.html
2013/4/29 不坏阿峰 <onlydebian at gmail.com>
> i have try put <pattern>vmkernel</pattern> <pattern>hostd-probe</pattern> do
> test, not work.
>
>
> 2013/4/29 Evan Rempel <erempel at uvic.ca>
>
>> That looks more like what I would expect.
>> In your example source line your $PROGRAM will be vmkernel and should be the text in the <pattern></pattern> xml tag.
>>
>> Aslo, your pattern needs to start at the text following the vmkernel: part of the syslog line. Only the $MESSAGE part of the syslog line is sent to the patterndb for parsing, unless your source definition in the syslog-ng.conf file has the flags(no-parse) option, but that would be unusual.
>>
>> Evan
>>
>>
>> Evan Rempel 250.271.7691
>> University Systems, University of Victoria
>>
>> 不坏阿峰 <onlydebian at gmail.com> wrote:
>>
>>
>> sorry for miss the purpose what i want to do.
>> (1) first . receive syslog from esxi host from UDP . (done)
>> (2) second. parse the log from UDP and parse with pattern db and get
>> separate imformation ( meet the problem i ask for help)
>> (3)third. store separate infor to Oracle table(done, test successfully
>> on syslog-ng macro value)
>>
>> for the second step, i use the way file() to check the situation of
>> db-parse.
>>
>> some sample log message from esxi host.
>>
>> Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP:
>> mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0)
>> Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default']
>> Unrecognized
>>
>>
>>
>>
>>
>> 2013/4/29 不坏阿峰 <onlydebian at gmail.com>
>>
>>> attachment is my current syslog-ng.conf. and esxi_pattern.xml.
>>>
>>> my syslog-ng receive UDP log from esxi host and try to test the
>>> db-parse and log it.
>>>
>>> i have change to <pattern>system</pattern>, but still can not get
>>> value from parse refer macro.
>>>
>>> thanks.
>>>
>>>
>>> 2013/4/28 Evan Rempel <erempel at uvic.ca>
>>>
>>>> Sorry for not being more clear in my first response.
>>>>
>>>> You have a template of
>>>>
>>>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time}
>>>> HOSTIP ${.esxi.host_ip},${.esxi.message}\n")
>>>>
>>>> When syslog-ng receives a syslog message, it logged it as;
>>>>
>>>> === system,error,critical, HOST IP ,
>>>>
>>>> This means that $PROGRAM contains "system"
>>>>
>>>> Now for the patterndb part.
>>>>
>>>> The patterndb parser FIRST matches $PROGRAM To the
>>>> <pattern>XXXX</pattern> in the <ruleset>
>>>>
>>>> <?xml version="1.0" encoding="utf-8"?>
>>>> <patterndb version='3' pub_date='2009-04-17'>
>>>> <ruleset name='esxi' id='123456678'>
>>>> <pattern>XXXX</pattern>
>>>>
>>>> In your case you have specified <pattern>ESXI</pattern> so the
>>>> patterndb parser will NOT use any
>>>> of your patterndb because it does not match the $PROGRAM
>>>>
>>>> You need to use
>>>>
>>>> ######## esxi_pattern.xml ############
>>>> <?xml version="1.0" encoding="utf-8"?>
>>>> <patterndb version='3' pub_date='2009-04-17'>
>>>> <ruleset name='esxi' id='123456678'>
>>>> <pattern>system</pattern>
>>>> <rules>
>>>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>>>> <patterns>
>>>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
>>>> @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>>>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>>>> </patterns>
>>>> </rule>
>>>> </rules>
>>>> </ruleset>
>>>> </patterndb>
>>>>
>>>>
>>>>
>>>> You have not included a complete syslong-ng source line for me to see
>>>> what you are trying to match against so I can
>>>> not tell if you pattern will actually match the lines that you are
>>>> trying to match.
>>>> At my organization we run ESX as well, and none of our lines would
>>>> match the pattern that you have, but
>>>> your environment might be different.
>>>>
>>>> I hope this was more clear.
>>>>
>>>> Evan.
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________________
>>>> From: 不坏阿峰 [onlydebian at gmail.com]
>>>> Sent: Sunday, April 28, 2013 8:24 AM
>>>> To: syslog-ng at lists.balabit.hu; Evan Rempel
>>>> Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13
>>>> debian squeeze)
>>>>
>>>> thanks to your reply. i do not understand how to do now. it puzzle and
>>>> trouble me some days. i read the balabit syslog-ng OSE guide documents
>>>> and only have simple information in there.
>>>>
>>>> how to do on this
>>>> ----->>>>
>>>> If you change the patterndb ruleset pattern to use a program of system
>>>> rather than ESXI I think it would work.
>>>>
>>>>
>>>> 2013/4/28 <syslog-ng-request at lists.balabit.hu<mailto:
>>>> syslog-ng-request at lists.balabit.hu>>
>>>> Send syslog-ng mailing list submissions to
>>>> syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
>>>>
>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>> or, via email, send a message with subject or body 'help' to
>>>> syslog-ng-request at lists.balabit.hu<mailto:
>>>> syslog-ng-request at lists.balabit.hu>
>>>>
>>>> You can reach the person managing the list at
>>>> syslog-ng-owner at lists.balabit.hu<mailto:
>>>> syslog-ng-owner at lists.balabit.hu>
>>>>
>>>> When replying, please edit your Subject line so it is more specific
>>>> than "Re: Contents of syslog-ng digest..."
>>>>
>>>>
>>>> Today's Topics:
>>>>
>>>> 1. Can not get DBParse match macro result (syslog-ng 3.13
>>>> debian squeeze) (????)
>>>> 2. Re: Can not get DBParse match macro result (syslog-ng 3.13
>>>> debian squeeze) (Evan Rempel)
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>>
>>>> Message: 1
>>>> Date: Sat, 27 Apr 2013 22:34:50 +0800
>>>> From: ???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com>>
>>>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng
>>>> 3.13 debian squeeze)
>>>> To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
>>>> Message-ID:
>>>> <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR=
>>>> V7wJ1EJdvE6Zx9zg at mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg at mail.gmail.com
>>>> >>
>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>
>>>> when use pdbtool do match test, it is success. but from syslog-ng can
>>>> not
>>>> return result of macro
>>>> i can not get macro result. for example, ${.esxi.month} no value,
>>>> same
>>>> as ${.esxi.host_ip} ${.esxi.time}
>>>>
>>>> test log output ,just like this.
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>>
>>>>
>>>> do the pdbtool test, it's ok. wish someone can give me some solution
>>>> and
>>>> help. i have search some mail list but i can not get the right solution.
>>>> thanks a lot.
>>>>
>>>> root at debian:~# pdbtool match -D -c -p
>>>> /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31
>>>> 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319:
>>>> Cmd
>>>> 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path
>>>> "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20
>>>> 0x0.
>>>> Act:NONE"
>>>> Pattern matching part:
>>>> @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@
>>>> @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@
>>>> @ESTRING:.esxi.program=
>>>> vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
>>>> Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP:
>>>> nmp_ThrottleLogForDevice:2319: Cmd
>>>> 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path
>>>> vmhba0:C0:T0:L0
>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
>>>> Matching part:
>>>> Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0
>>>> Valid
>>>> sense data: 0x5 0x20 0x0. Act:NONE
>>>> Values:
>>>> MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0
>>>> Valid
>>>> sense data: 0x5 0x20 0x0. Act:NONE
>>>> PROGRAM=ESXI
>>>> .classifier.class=esxi
>>>> .classifier.rule_id=182437592347598
>>>> .esxi.month=Apr
>>>> .esxi.date=26
>>>> .esxi.time=15:17:31
>>>> .esxi.host_ip=192.168.88.71
>>>> .esxi.program= vmkernel
>>>> .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
>>>> root at debian:~#
>>>>
>>>>
>>>> my configuration like as below
>>>>
>>>> ######## esxi_pattern.xml ############
>>>> <?xml version="1.0" encoding="utf-8"?>
>>>> <patterndb version='3' pub_date='2009-04-17'>
>>>> <ruleset name='esxi' id='123456678'>
>>>> <pattern>ESXI</pattern>
>>>> <rules>
>>>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>>>> <patterns>
>>>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
>>>> @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>>>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>>>> </patterns>
>>>> </rule>
>>>> </rules>
>>>> </ruleset>
>>>> </patterndb>
>>>>
>>>> ######## syslog-ng.conf ########
>>>>
>>>> #####Parser#####
>>>> parser pattern_db {
>>>> db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
>>>> };
>>>>
>>>> #Check pattern matching
>>>> destination udp_esxi_output {
>>>> file("/var/log/pattern_output"
>>>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time}
>>>> HOST
>>>> IP ${.esxi.host_ip},${.esxi.message}\n")
>>>> template_escape(no));
>>>> };
>>>>
>>>> #####Log#####
>>>> log {
>>>> source(s_network);
>>>> parser(pattern_db);
>>>> destination(udp_esxi_output);
>>>> };
>>>> -------------- next part --------------
>>>> An HTML attachment was scrubbed...
>>>> URL:
>>>> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/20e80756/attachment.html
>>>>
>>>> ------------------------------
>>>>
>>>> Message: 2
>>>> Date: Sat, 27 Apr 2013 16:10:02 +0000
>>>> From: Evan Rempel <erempel at uvic.ca<mailto:erempel at uvic.ca>>
>>>> Subject: Re: [syslog-ng] Can not get DBParse match macro result
>>>> (syslog-ng 3.13 debian squeeze)
>>>> To: "Syslog-ng users' and developers' mailing list"
>>>> <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu
>>>> >>
>>>> Message-ID: <q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com
>>>> <mailto:q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com>>
>>>> Content-Type: text/plain; charset="iso-2022-jp"
>>>>
>>>> It would appear that you have everything correct when the "PROGRAM" is
>>>> ESXI but the log line as syslog-ng sees it has a PROGRAM of "system"
>>>> according to your test log output.
>>>>
>>>> If you change the patterndb ruleset pattern to use a program of system
>>>> rather than ESXI I think it would work.
>>>>
>>>>
>>>> Evan Rempel 250.271.7691<tel:250.271.7691>
>>>> University Systems, University of Victoria
>>>>
>>>> ???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com>> wrote:
>>>>
>>>>
>>>>
>>>> when use pdbtool do match test, it is success. but from syslog-ng can
>>>> not return result of macro
>>>> i can not get macro result. for example, ${.esxi.month} no value,
>>>> same as ${.esxi.host_ip} ${.esxi.time}
>>>>
>>>> test log output ,just like this.
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>> === system,error,critical, HOST IP ,
>>>>
>>>>
>>>> do the pdbtool test, it's ok. wish someone can give me some solution
>>>> and help. i have search some mail list but i can not get the right
>>>> solution. thanks a lot.
>>>>
>>>> root at debian:~# pdbtool match -D -c -p
>>>> /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31
>>>> 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
>>>> 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path
>>>> "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
>>>> Act:NONE"
>>>> Pattern matching part:
>>>> @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@@STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program=
>>>> vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
>>>> 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
>>>> Matching part:
>>>> Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
>>>> sense data: 0x5 0x20 0x0. Act:NONE
>>>> Values:
>>>> MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
>>>> sense data: 0x5 0x20 0x0. Act:NONE
>>>> PROGRAM=ESXI
>>>> .classifier.class=esxi
>>>> .classifier.rule_id=182437592347598
>>>> .esxi.month=Apr
>>>> .esxi.date=26
>>>> .esxi.time=15:17:31
>>>> .esxi.host_ip=192.168.88.71
>>>> .esxi.program= vmkernel
>>>> .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
>>>> root at debian:~#
>>>>
>>>>
>>>> my configuration like as below
>>>>
>>>> ######## esxi_pattern.xml ############
>>>> <?xml version="1.0" encoding="utf-8"?>
>>>> <patterndb version='3' pub_date='2009-04-17'>
>>>> <ruleset name='esxi' id='123456678'>
>>>> <pattern>ESXI</pattern>
>>>> <rules>
>>>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>>>> <patterns>
>>>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@@STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>>>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>>>> </patterns>
>>>> </rule>
>>>> </rules>
>>>> </ruleset>
>>>> </patterndb>
>>>>
>>>> ######## syslog-ng.conf ########
>>>>
>>>> #####Parser#####
>>>> parser pattern_db {
>>>> db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
>>>> };
>>>>
>>>> #Check pattern matching
>>>> destination udp_esxi_output {
>>>> file("/var/log/pattern_output"
>>>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time}
>>>> HOST IP ${.esxi.host_ip},${.esxi.message}\n")
>>>> template_escape(no));
>>>> };
>>>>
>>>> #####Log#####
>>>> log {
>>>> source(s_network);
>>>> parser(pattern_db);
>>>> destination(udp_esxi_output);
>>>> };
>>>>
>>>> -------------- next part --------------
>>>> An HTML attachment was scrubbed...
>>>> URL:
>>>> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/2f67c039/attachment-0001.htm
>>>>
>>>> ------------------------------
>>>>
>>>> _______________________________________________
>>>> syslog-ng maillist - syslog-ng at lists.balabit.hu<mailto:
>>>> syslog-ng at lists.balabit.hu>
>>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>
>>>>
>>>> End of syslog-ng Digest, Vol 96, Issue 25
>>>> *****************************************
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130429/b2245c4d/attachment-0001.htm
More information about the syslog-ng
mailing list