[syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
Martin Holste
mcholste at gmail.com
Mon May 6 03:26:08 CEST 2013
The issue is that your test message is actually the raw message as seen in
a log file containing the timestamp, host, and program as well as the
message. In the syslog-ng.conf, that would correspond to $S_TIMESTAMP,
$HOST, $PROGRAM, and $MSG_ONLY. Your pattern isn't actually extracting any
fields not already extracted by syslog-ng without the patterndb. Are you
collecting this file using the file() driver or are you reading from the
network? If this is a raw file you are reading, then there is no program,
and so you just need to remove the first <pattern> element entirely to say
to match on all programs, or use the program_override() flag to set a
program explicitly and make sure it matches the program in the <pattern>
element.
On Mon, Apr 29, 2013 at 4:35 AM, 不坏阿峰 <onlydebian at gmail.com> wrote:
> Dear Marton, Martin
>
> i saw your post in mail list. could you give me some advice to solve my
> problem. why i can not get db-parse macro value from syslog-ng while
> pdbtool match work successfully.
>
> thanks.
>
> my thread is here
>
> https://lists.balabit.hu/pipermail/syslog-ng/2013-April/020300.html
>
>
> 2013/4/29 不坏阿峰 <onlydebian at gmail.com>
>
>> i have try put <pattern>vmkernel</pattern> <pattern>hostd-probe</pattern>
>> do test, not work.
>>
>>
>> 2013/4/29 Evan Rempel <erempel at uvic.ca>
>>
>>> That looks more like what I would expect.
>>> In your example source line your $PROGRAM will be vmkernel and should be the text in the <pattern></pattern> xml tag.
>>>
>>> Aslo, your pattern needs to start at the text following the vmkernel: part of the syslog line. Only the $MESSAGE part of the syslog line is sent to the patterndb for parsing, unless your source definition in the syslog-ng.conf file has the flags(no-parse) option, but that would be unusual.
>>>
>>> Evan
>>>
>>>
>>> Evan Rempel 250.271.7691
>>> University Systems, University of Victoria
>>>
>>> 不坏阿峰 <onlydebian at gmail.com> wrote:
>>>
>>>
>>> sorry for miss the purpose what i want to do.
>>> (1) first . receive syslog from esxi host from UDP . (done)
>>> (2) second. parse the log from UDP and parse with pattern db and get
>>> separate imformation ( meet the problem i ask for help)
>>> (3)third. store separate infor to Oracle table(done, test successfully
>>> on syslog-ng macro value)
>>>
>>> for the second step, i use the way file() to check the situation of
>>> db-parse.
>>>
>>> some sample log message from esxi host.
>>>
>>> Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP:
>>> mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0)
>>> Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default']
>>> Unrecognized
>>>
>>>
>>>
>>>
>>>
>>> 2013/4/29 不坏阿峰 <onlydebian at gmail.com>
>>>
>>>> attachment is my current syslog-ng.conf. and esxi_pattern.xml.
>>>>
>>>> my syslog-ng receive UDP log from esxi host and try to test the
>>>> db-parse and log it.
>>>>
>>>> i have change to <pattern>system</pattern>, but still can not get
>>>> value from parse refer macro.
>>>>
>>>> thanks.
>>>>
>>>>
>>>> 2013/4/28 Evan Rempel <erempel at uvic.ca>
>>>>
>>>>> Sorry for not being more clear in my first response.
>>>>>
>>>>> You have a template of
>>>>>
>>>>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time}
>>>>> HOSTIP ${.esxi.host_ip},${.esxi.message}\n")
>>>>>
>>>>> When syslog-ng receives a syslog message, it logged it as;
>>>>>
>>>>> === system,error,critical, HOST IP ,
>>>>>
>>>>> This means that $PROGRAM contains "system"
>>>>>
>>>>> Now for the patterndb part.
>>>>>
>>>>> The patterndb parser FIRST matches $PROGRAM To the
>>>>> <pattern>XXXX</pattern> in the <ruleset>
>>>>>
>>>>> <?xml version="1.0" encoding="utf-8"?>
>>>>> <patterndb version='3' pub_date='2009-04-17'>
>>>>> <ruleset name='esxi' id='123456678'>
>>>>> <pattern>XXXX</pattern>
>>>>>
>>>>> In your case you have specified <pattern>ESXI</pattern> so the
>>>>> patterndb parser will NOT use any
>>>>> of your patterndb because it does not match the $PROGRAM
>>>>>
>>>>> You need to use
>>>>>
>>>>> ######## esxi_pattern.xml ############
>>>>> <?xml version="1.0" encoding="utf-8"?>
>>>>> <patterndb version='3' pub_date='2009-04-17'>
>>>>> <ruleset name='esxi' id='123456678'>
>>>>> <pattern>system</pattern>
>>>>> <rules>
>>>>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>>>>> <patterns>
>>>>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
>>>>> @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>>>>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>>>>> </patterns>
>>>>> </rule>
>>>>> </rules>
>>>>> </ruleset>
>>>>> </patterndb>
>>>>>
>>>>>
>>>>>
>>>>> You have not included a complete syslong-ng source line for me to see
>>>>> what you are trying to match against so I can
>>>>> not tell if you pattern will actually match the lines that you are
>>>>> trying to match.
>>>>> At my organization we run ESX as well, and none of our lines would
>>>>> match the pattern that you have, but
>>>>> your environment might be different.
>>>>>
>>>>> I hope this was more clear.
>>>>>
>>>>> Evan.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ________________________________________
>>>>> From: 不坏阿峰 [onlydebian at gmail.com]
>>>>> Sent: Sunday, April 28, 2013 8:24 AM
>>>>> To: syslog-ng at lists.balabit.hu; Evan Rempel
>>>>> Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13
>>>>> debian squeeze)
>>>>>
>>>>> thanks to your reply. i do not understand how to do now. it puzzle
>>>>> and trouble me some days. i read the balabit syslog-ng OSE guide
>>>>> documents and only have simple information in there.
>>>>>
>>>>> how to do on this
>>>>> ----->>>>
>>>>> If you change the patterndb ruleset pattern to use a program of system
>>>>> rather than ESXI I think it would work.
>>>>>
>>>>>
>>>>> 2013/4/28 <syslog-ng-request at lists.balabit.hu<mailto:
>>>>> syslog-ng-request at lists.balabit.hu>>
>>>>> Send syslog-ng mailing list submissions to
>>>>> syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
>>>>>
>>>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>> or, via email, send a message with subject or body 'help' to
>>>>> syslog-ng-request at lists.balabit.hu<mailto:
>>>>> syslog-ng-request at lists.balabit.hu>
>>>>>
>>>>> You can reach the person managing the list at
>>>>> syslog-ng-owner at lists.balabit.hu<mailto:
>>>>> syslog-ng-owner at lists.balabit.hu>
>>>>>
>>>>> When replying, please edit your Subject line so it is more specific
>>>>> than "Re: Contents of syslog-ng digest..."
>>>>>
>>>>>
>>>>> Today's Topics:
>>>>>
>>>>> 1. Can not get DBParse match macro result (syslog-ng 3.13
>>>>> debian squeeze) (????)
>>>>> 2. Re: Can not get DBParse match macro result (syslog-ng 3.13
>>>>> debian squeeze) (Evan Rempel)
>>>>>
>>>>>
>>>>> ----------------------------------------------------------------------
>>>>>
>>>>> Message: 1
>>>>> Date: Sat, 27 Apr 2013 22:34:50 +0800
>>>>> From: ???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com>>
>>>>> Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng
>>>>> 3.13 debian squeeze)
>>>>> To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>
>>>>> Message-ID:
>>>>> <CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR=
>>>>> V7wJ1EJdvE6Zx9zg at mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg at mail.gmail.com
>>>>> >>
>>>>> Content-Type: text/plain; charset="iso-8859-1"
>>>>>
>>>>> when use pdbtool do match test, it is success. but from syslog-ng can
>>>>> not
>>>>> return result of macro
>>>>> i can not get macro result. for example, ${.esxi.month} no value,
>>>>> same
>>>>> as ${.esxi.host_ip} ${.esxi.time}
>>>>>
>>>>> test log output ,just like this.
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>>
>>>>>
>>>>> do the pdbtool test, it's ok. wish someone can give me some solution
>>>>> and
>>>>> help. i have search some mail list but i can not get the right
>>>>> solution.
>>>>> thanks a lot.
>>>>>
>>>>> root at debian:~# pdbtool match -D -c -p
>>>>> /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31
>>>>> 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319:
>>>>> Cmd
>>>>> 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path
>>>>> "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20
>>>>> 0x0.
>>>>> Act:NONE"
>>>>> Pattern matching part:
>>>>> @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@
>>>>> @STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@
>>>>> @ESTRING:.esxi.program=
>>>>> vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
>>>>> Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP:
>>>>> nmp_ThrottleLogForDevice:2319: Cmd
>>>>> 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path
>>>>> vmhba0:C0:T0:L0
>>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
>>>>> Matching part:
>>>>> Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0
>>>>> Valid
>>>>> sense data: 0x5 0x20 0x0. Act:NONE
>>>>> Values:
>>>>> MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0
>>>>> Valid
>>>>> sense data: 0x5 0x20 0x0. Act:NONE
>>>>> PROGRAM=ESXI
>>>>> .classifier.class=esxi
>>>>> .classifier.rule_id=182437592347598
>>>>> .esxi.month=Apr
>>>>> .esxi.date=26
>>>>> .esxi.time=15:17:31
>>>>> .esxi.host_ip=192.168.88.71
>>>>> .esxi.program= vmkernel
>>>>> .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
>>>>> root at debian:~#
>>>>>
>>>>>
>>>>> my configuration like as below
>>>>>
>>>>> ######## esxi_pattern.xml ############
>>>>> <?xml version="1.0" encoding="utf-8"?>
>>>>> <patterndb version='3' pub_date='2009-04-17'>
>>>>> <ruleset name='esxi' id='123456678'>
>>>>> <pattern>ESXI</pattern>
>>>>> <rules>
>>>>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>>>>> <patterns>
>>>>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
>>>>> @STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>>>>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>>>>> </patterns>
>>>>> </rule>
>>>>> </rules>
>>>>> </ruleset>
>>>>> </patterndb>
>>>>>
>>>>> ######## syslog-ng.conf ########
>>>>>
>>>>> #####Parser#####
>>>>> parser pattern_db {
>>>>> db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
>>>>> };
>>>>>
>>>>> #Check pattern matching
>>>>> destination udp_esxi_output {
>>>>> file("/var/log/pattern_output"
>>>>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time}
>>>>> HOST
>>>>> IP ${.esxi.host_ip},${.esxi.message}\n")
>>>>> template_escape(no));
>>>>> };
>>>>>
>>>>> #####Log#####
>>>>> log {
>>>>> source(s_network);
>>>>> parser(pattern_db);
>>>>> destination(udp_esxi_output);
>>>>> };
>>>>> -------------- next part --------------
>>>>> An HTML attachment was scrubbed...
>>>>> URL:
>>>>> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/20e80756/attachment.html
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Message: 2
>>>>> Date: Sat, 27 Apr 2013 16:10:02 +0000
>>>>> From: Evan Rempel <erempel at uvic.ca<mailto:erempel at uvic.ca>>
>>>>> Subject: Re: [syslog-ng] Can not get DBParse match macro result
>>>>> (syslog-ng 3.13 debian squeeze)
>>>>> To: "Syslog-ng users' and developers' mailing list"
>>>>> <syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu
>>>>> >>
>>>>> Message-ID: <q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com
>>>>> <mailto:q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com>>
>>>>> Content-Type: text/plain; charset="iso-2022-jp"
>>>>>
>>>>> It would appear that you have everything correct when the "PROGRAM" is
>>>>> ESXI but the log line as syslog-ng sees it has a PROGRAM of "system"
>>>>> according to your test log output.
>>>>>
>>>>> If you change the patterndb ruleset pattern to use a program of system
>>>>> rather than ESXI I think it would work.
>>>>>
>>>>>
>>>>> Evan Rempel 250.271.7691<tel:250.271.7691>
>>>>> University Systems, University of Victoria
>>>>>
>>>>> ???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> when use pdbtool do match test, it is success. but from syslog-ng can
>>>>> not return result of macro
>>>>> i can not get macro result. for example, ${.esxi.month} no value,
>>>>> same as ${.esxi.host_ip} ${.esxi.time}
>>>>>
>>>>> test log output ,just like this.
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>> === system,error,critical, HOST IP ,
>>>>>
>>>>>
>>>>> do the pdbtool test, it's ok. wish someone can give me some solution
>>>>> and help. i have search some mail list but i can not get the right
>>>>> solution. thanks a lot.
>>>>>
>>>>> root at debian:~# pdbtool match -D -c -p
>>>>> /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31
>>>>> 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
>>>>> 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path
>>>>> "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
>>>>> Act:NONE"
>>>>> Pattern matching part:
>>>>> @STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@@STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program=
>>>>> vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
>>>>> 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
>>>>> Matching part:
>>>>> Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
>>>>> sense data: 0x5 0x20 0x0. Act:NONE
>>>>> Values:
>>>>> MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
>>>>> nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
>>>>> mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
>>>>> sense data: 0x5 0x20 0x0. Act:NONE
>>>>> PROGRAM=ESXI
>>>>> .classifier.class=esxi
>>>>> .classifier.rule_id=182437592347598
>>>>> .esxi.month=Apr
>>>>> .esxi.date=26
>>>>> .esxi.time=15:17:31
>>>>> .esxi.host_ip=192.168.88.71
>>>>> .esxi.program= vmkernel
>>>>> .esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
>>>>> (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
>>>>> Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
>>>>> root at debian:~#
>>>>>
>>>>>
>>>>> my configuration like as below
>>>>>
>>>>> ######## esxi_pattern.xml ############
>>>>> <?xml version="1.0" encoding="utf-8"?>
>>>>> <patterndb version='3' pub_date='2009-04-17'>
>>>>> <ruleset name='esxi' id='123456678'>
>>>>> <pattern>ESXI</pattern>
>>>>> <rules>
>>>>> <rule provider='Fone Bro' id='182437592347598' class='esxi'>
>>>>> <patterns>
>>>>> <pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@@STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
>>>>> @ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
>>>>> </patterns>
>>>>> </rule>
>>>>> </rules>
>>>>> </ruleset>
>>>>> </patterndb>
>>>>>
>>>>> ######## syslog-ng.conf ########
>>>>>
>>>>> #####Parser#####
>>>>> parser pattern_db {
>>>>> db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
>>>>> };
>>>>>
>>>>> #Check pattern matching
>>>>> destination udp_esxi_output {
>>>>> file("/var/log/pattern_output"
>>>>> template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time}
>>>>> HOST IP ${.esxi.host_ip},${.esxi.message}\n")
>>>>> template_escape(no));
>>>>> };
>>>>>
>>>>> #####Log#####
>>>>> log {
>>>>> source(s_network);
>>>>> parser(pattern_db);
>>>>> destination(udp_esxi_output);
>>>>> };
>>>>>
>>>>> -------------- next part --------------
>>>>> An HTML attachment was scrubbed...
>>>>> URL:
>>>>> http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/2f67c039/attachment-0001.htm
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> syslog-ng maillist - syslog-ng at lists.balabit.hu<mailto:
>>>>> syslog-ng at lists.balabit.hu>
>>>>> https://lists.balabit.hu/mailman/listinfo/syslog-ng
>>>>>
>>>>>
>>>>> End of syslog-ng Digest, Vol 96, Issue 25
>>>>> *****************************************
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130505/48250535/attachment-0001.htm
More information about the syslog-ng
mailing list