<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
</head>
<body>
<pre style="word-wrap:break-word; font-size:10.0pt; font-family:Tahoma; color:black">That looks more like what I would expect.
In your example source line your $PROGRAM will be vmkernel and should be the text in the <pattern></pattern> xml tag.
Aslo, your pattern needs to start at the text following the vmkernel: part of the syslog line. Only the $MESSAGE part of the syslog line is sent to the patterndb for parsing, unless your source definition in the syslog-ng.conf file has the flags(no-parse) option, but that would be unusual.
Evan
Evan Rempel 250.271.7691
University Systems, University of Victoria
²»»µ°¢·å <onlydebian@gmail.com> wrote:
</pre>
<div>
<div dir="ltr">sorry for miss the purpose what i want to do.
<div>(1) first . receive syslog from esxi host from UDP . (done)</div>
<div>(2) second. parse the log from UDP and parse with pattern db and get separate imformation ( meet the problem i ask for help)</div>
<div>(3)third. store separate infor to Oracle table(done, test successfully on syslog-ng macro value)</div>
<div><br>
</div>
<div style="">for the second step, i use the way file() to check the situation of db-parse.</div>
<div><br>
</div>
<div>some sample log message from esxi host.<br>
<div>
<div><br>
</div>
<div>
<div>Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP: mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0) </div>
<div>Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default'] Unrecognized </div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2013/4/29 ²»»µ°¢·å <span dir="ltr"><<a href="mailto:onlydebian@gmail.com" target="_blank">onlydebian@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">attachment is my current syslog-ng.conf. and esxi_pattern.xml.
<div><br>
</div>
<div>my syslog-ng receive UDP log from esxi host and try to test the db-parse and log it.</div>
<div><br>
</div>
<div>i have change to <span style="font-family:arial,sans-serif; font-size:14px"><pattern>system</pattern>, but still can not get value from parse refer macro.</span></div>
<div><span style="font-family:arial,sans-serif; font-size:14px"><br>
</span></div>
<div><font face="arial, sans-serif"><span style="font-size:14px">thanks.</span></font></div>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2013/4/28 Evan Rempel <span dir="ltr"><<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
Sorry for not being more clear in my first response.<br>
<br>
You have a template of<br>
<br>
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP ${.esxi.host_ip},${.esxi.message}\n")<br>
<br>
When syslog-ng receives a syslog message, it logged it as;<br>
<div><br>
=== system,error,critical, HOST IP ,<br>
<br>
</div>
This means that $PROGRAM contains "system"<br>
<br>
Now for the patterndb part.<br>
<br>
The patterndb parser FIRST matches $PROGRAM To the <pattern>XXXX</pattern> in the <ruleset><br>
<div><br>
<?xml version="1.0" encoding="utf-8"?><br>
<patterndb version='3' pub_date='2009-04-17'><br>
<ruleset name='esxi' id='123456678'><br>
</div>
<pattern>XXXX</pattern><br>
<br>
In your case you have specified <pattern>ESXI</pattern> so the patterndb parser will NOT use any<br>
of your patterndb because it does not match the $PROGRAM<br>
<br>
You need to use<br>
<div><br>
######## esxi_pattern.xml ############<br>
<?xml version="1.0" encoding="utf-8"?><br>
<patterndb version='3' pub_date='2009-04-17'><br>
<ruleset name='esxi' id='123456678'><br>
</div>
<pattern>system</pattern><br>
<div><rules><br>
<rule provider='Fone Bro' id='182437592347598' class='esxi'><br>
<patterns><br>
<pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@<br>
@STRING:.esxi.time::@@IPv4:.esxi.host_ip:@<br>
@ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern><br>
</patterns><br>
</rule><br>
</rules><br>
</ruleset><br>
</patterndb><br>
<br>
<br>
<br>
</div>
You have not included a complete syslong-ng source line for me to see what you are trying to match against so I can<br>
not tell if you pattern will actually match the lines that you are trying to match.<br>
At my organization we run ESX as well, and none of our lines would match the pattern that you have, but<br>
your environment might be different.<br>
<br>
I hope this was more clear.<br>
<br>
Evan.<br>
<br>
<br>
<br>
<br>
________________________________________<br>
From: ²»»µ°¢·å [<a href="mailto:onlydebian@gmail.com" target="_blank">onlydebian@gmail.com</a>]<br>
Sent: Sunday, April 28, 2013 8:24 AM<br>
To: <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>; Evan Rempel<br>
Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)<br>
<div><br>
thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there.<br>
<br>
how to do on this<br>
----->>>><br>
If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work.<br>
<br>
<br>
</div>
2013/4/28 <<a href="mailto:syslog-ng-request@lists.balabit.hu" target="_blank">syslog-ng-request@lists.balabit.hu</a><mailto:<a href="mailto:syslog-ng-request@lists.balabit.hu" target="_blank">syslog-ng-request@lists.balabit.hu</a>>><br>
<div>Send syslog-ng mailing list submissions to<br>
</div>
<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><mailto:<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
<div><br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">
https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
or, via email, send a message with subject or body 'help' to<br>
</div>
<a href="mailto:syslog-ng-request@lists.balabit.hu" target="_blank">syslog-ng-request@lists.balabit.hu</a><mailto:<a href="mailto:syslog-ng-request@lists.balabit.hu" target="_blank">syslog-ng-request@lists.balabit.hu</a>><br>
<div><br>
You can reach the person managing the list at<br>
</div>
<a href="mailto:syslog-ng-owner@lists.balabit.hu" target="_blank">syslog-ng-owner@lists.balabit.hu</a><mailto:<a href="mailto:syslog-ng-owner@lists.balabit.hu" target="_blank">syslog-ng-owner@lists.balabit.hu</a>><br>
<div><br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of syslog-ng digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Can not get DBParse match macro result (syslog-ng 3.13<br>
debian squeeze) (????)<br>
2. Re: Can not get DBParse match macro result (syslog-ng 3.13<br>
debian squeeze) (Evan Rempel)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sat, 27 Apr 2013 22:34:50 +0800<br>
</div>
From: ???? <<a href="mailto:onlydebian@gmail.com" target="_blank">onlydebian@gmail.com</a><mailto:<a href="mailto:onlydebian@gmail.com" target="_blank">onlydebian@gmail.com</a>>><br>
<div>Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng<br>
3.13 debian squeeze)<br>
</div>
To: <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><mailto:<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
Message-ID:<br>
<CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR=<a href="mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com" target="_blank">V7wJ1EJdvE6Zx9zg@mail.gmail.com</a><mailto:<a href="mailto:V7wJ1EJdvE6Zx9zg@mail.gmail.com" target="_blank">V7wJ1EJdvE6Zx9zg@mail.gmail.com</a>>><br>
<div>
<div>Content-Type: text/plain; charset="iso-8859-1"<br>
<br>
when use pdbtool do match test, it is success. but from syslog-ng can not<br>
return result of macro<br>
i can not get macro result. for example, ${.esxi.month} no value, same<br>
as ${.esxi.host_ip} ${.esxi.time}<br>
<br>
test log output ,just like this.<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
<br>
<br>
do the pdbtool test, it's ok. wish someone can give me some solution and<br>
help. i have search some mail list but i can not get the right solution.<br>
thanks a lot.<br>
<br>
root@debian:~# pdbtool match -D -c -p<br>
/etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31<br>
192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd<br>
0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path<br>
"vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.<br>
Act:NONE"<br>
Pattern matching part:<br>
@STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@<br>
@STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program=<br>
vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a<br>
(0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0<br>
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.<br>
Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP:<br>
nmp_ThrottleLogForDevice:2319: Cmd<br>
0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0<br>
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@<br>
Matching part:<br>
Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:<br>
nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev<br>
mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid<br>
sense data: 0x5 0x20 0x0. Act:NONE<br>
Values:<br>
MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:<br>
nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev<br>
mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid<br>
sense data: 0x5 0x20 0x0. Act:NONE<br>
PROGRAM=ESXI<br>
.classifier.class=esxi<br>
.classifier.rule_id=182437592347598<br>
.esxi.month=Apr<br>
.esxi.date=26<br>
.esxi.time=15:17:31<br>
.esxi.host_ip=192.168.88.71<br>
.esxi.program= vmkernel<br>
.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a<br>
(0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0<br>
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE<br>
root@debian:~#<br>
<br>
<br>
my configuration like as below<br>
<br>
######## esxi_pattern.xml ############<br>
<?xml version="1.0" encoding="utf-8"?><br>
<patterndb version='3' pub_date='2009-04-17'><br>
<ruleset name='esxi' id='123456678'><br>
<pattern>ESXI</pattern><br>
<rules><br>
<rule provider='Fone Bro' id='182437592347598' class='esxi'><br>
<patterns><br>
<pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@<br>
@STRING:.esxi.time::@@IPv4:.esxi.host_ip:@<br>
@ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern><br>
</patterns><br>
</rule><br>
</rules><br>
</ruleset><br>
</patterndb><br>
<br>
######## syslog-ng.conf ########<br>
<br>
#####Parser#####<br>
parser pattern_db {<br>
db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));<br>
};<br>
<br>
#Check pattern matching<br>
destination udp_esxi_output {<br>
file("/var/log/pattern_output"<br>
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST<br>
IP ${.esxi.host_ip},${.esxi.message}\n")<br>
template_escape(no));<br>
};<br>
<br>
#####Log#####<br>
log {<br>
source(s_network);<br>
parser(pattern_db);<br>
destination(udp_esxi_output);<br>
};<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/20e80756/attachment.html" target="_blank">
http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/20e80756/attachment.html</a><br>
<br>
------------------------------<br>
<br>
Message: 2<br>
Date: Sat, 27 Apr 2013 16:10:02 +0000<br>
</div>
</div>
From: Evan Rempel <<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a><mailto:<a href="mailto:erempel@uvic.ca" target="_blank">erempel@uvic.ca</a>>><br>
<div>Subject: Re: [syslog-ng] Can not get DBParse match macro result<br>
(syslog-ng 3.13 debian squeeze)<br>
To: "Syslog-ng users' and developers' mailing list"<br>
</div>
<<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><mailto:<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>>><br>
Message-ID: <<a href="mailto:q8vb966l4qe0219lsusm5ju4.1367078999747@email.android.com" target="_blank">q8vb966l4qe0219lsusm5ju4.1367078999747@email.android.com</a><mailto:<a href="mailto:q8vb966l4qe0219lsusm5ju4.1367078999747@email.android.com" target="_blank">q8vb966l4qe0219lsusm5ju4.1367078999747@email.android.com</a>>><br>
<div>Content-Type: text/plain; charset="iso-2022-jp"<br>
<br>
It would appear that you have everything correct when the "PROGRAM" is ESXI but the log line as syslog-ng sees it has a PROGRAM of "system" according to your test log output.<br>
<br>
If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work.<br>
<br>
<br>
</div>
Evan Rempel <a href="tel:250.271.7691" value="+12502717691" target="_blank">250.271.7691</a><tel:<a href="tel:250.271.7691" value="+12502717691" target="_blank">250.271.7691</a>><br>
<div>University Systems, University of Victoria<br>
<br>
</div>
<div>
<div>???? <<a href="mailto:onlydebian@gmail.com" target="_blank">onlydebian@gmail.com</a><mailto:<a href="mailto:onlydebian@gmail.com" target="_blank">onlydebian@gmail.com</a>>> wrote:<br>
<br>
<br>
<br>
when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro<br>
i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time}<br>
<br>
test log output ,just like this.<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
=== system,error,critical, HOST IP ,<br>
<br>
<br>
do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot.<br>
<br>
root@debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0"
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE"<br>
Pattern matching part:<br>
@STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@ @IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ @ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense
data: 0x5 0x20 0x0. Act:NONE@<br>
Matching part:<br>
Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE<br>
Values:<br>
MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE<br>
PROGRAM=ESXI<br>
.classifier.class=esxi<br>
.classifier.rule_id=182437592347598<br>
.esxi.month=Apr<br>
.esxi.date=26<br>
.esxi.time=15:17:31<br>
.esxi.host_ip=192.168.88.71<br>
.esxi.program= vmkernel<br>
.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE<br>
root@debian:~#<br>
<br>
<br>
my configuration like as below<br>
<br>
######## esxi_pattern.xml ############<br>
<?xml version="1.0" encoding="utf-8"?><br>
<patterndb version='3' pub_date='2009-04-17'><br>
<ruleset name='esxi' id='123456678'><br>
<pattern>ESXI</pattern><br>
<rules><br>
<rule provider='Fone Bro' id='182437592347598' class='esxi'><br>
<patterns><br>
<pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@ @IPv4:.esxi.host_ip:@@ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern><br>
</patterns><br>
</rule><br>
</rules><br>
</ruleset><br>
</patterndb><br>
<br>
######## syslog-ng.conf ########<br>
<br>
#####Parser#####<br>
parser pattern_db {<br>
db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));<br>
};<br>
<br>
#Check pattern matching<br>
destination udp_esxi_output {<br>
file("/var/log/pattern_output"<br>
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n")<br>
template_escape(no));<br>
};<br>
<br>
#####Log#####<br>
log {<br>
source(s_network);<br>
parser(pattern_db);<br>
destination(udp_esxi_output);<br>
};<br>
<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <a href="http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/2f67c039/attachment-0001.htm" target="_blank">
http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/2f67c039/attachment-0001.htm</a><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
</div>
</div>
syslog-ng maillist - <a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a><mailto:<a href="mailto:syslog-ng@lists.balabit.hu" target="_blank">syslog-ng@lists.balabit.hu</a>><br>
<div>
<div><a href="https://lists.balabit.hu/mailman/listinfo/syslog-ng" target="_blank">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a><br>
<br>
<br>
End of syslog-ng Digest, Vol 96, Issue 25<br>
*****************************************<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>