[syslog-ng] Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
Evan Rempel
erempel at uvic.ca
Sun Apr 28 18:32:38 CEST 2013
That looks more like what I would expect.
In your example source line your $PROGRAM will be vmkernel and should be the text in the <pattern></pattern> xml tag.
Aslo, your pattern needs to start at the text following the vmkernel: part of the syslog line. Only the $MESSAGE part of the syslog line is sent to the patterndb for parsing, unless your source definition in the syslog-ng.conf file has the flags(no-parse) option, but that would be unusual.
Evan
Evan Rempel 250.271.7691
University Systems, University of Victoria
不坏阿峰 <onlydebian at gmail.com> wrote:
sorry for miss the purpose what i want to do.
(1) first . receive syslog from esxi host from UDP . (done)
(2) second. parse the log from UDP and parse with pattern db and get separate imformation ( meet the problem i ask for help)
(3)third. store separate infor to Oracle table(done, test successfully on syslog-ng macro value)
for the second step, i use the way file() to check the situation of db-parse.
some sample log message from esxi host.
Apr 29 00:08:50 192.168.88.81 vmkernel: cpu6:10283)NMP: mp_ThrottleLogForDevice:2319: Cmd 0x1a (0x412400404280, 0)
Apr 29 00:10:02 192.168.88.81 hostd-probe: [FF9E8CB0 warning 'Default'] Unrecognized
2013/4/29 不坏阿峰 <onlydebian at gmail.com<mailto:onlydebian at gmail.com>>
attachment is my current syslog-ng.conf. and esxi_pattern.xml.
my syslog-ng receive UDP log from esxi host and try to test the db-parse and log it.
i have change to <pattern>system</pattern>, but still can not get value from parse refer macro.
thanks.
2013/4/28 Evan Rempel <erempel at uvic.ca<mailto:erempel at uvic.ca>>
Sorry for not being more clear in my first response.
You have a template of
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOSTIP ${.esxi.host_ip},${.esxi.message}\n")
When syslog-ng receives a syslog message, it logged it as;
=== system,error,critical, HOST IP ,
This means that $PROGRAM contains "system"
Now for the patterndb part.
The patterndb parser FIRST matches $PROGRAM To the <pattern>XXXX</pattern> in the <ruleset>
<?xml version="1.0" encoding="utf-8"?>
<patterndb version='3' pub_date='2009-04-17'>
<ruleset name='esxi' id='123456678'>
<pattern>XXXX</pattern>
In your case you have specified <pattern>ESXI</pattern> so the patterndb parser will NOT use any
of your patterndb because it does not match the $PROGRAM
You need to use
######## esxi_pattern.xml ############
<?xml version="1.0" encoding="utf-8"?>
<patterndb version='3' pub_date='2009-04-17'>
<ruleset name='esxi' id='123456678'>
<pattern>system</pattern>
<rules>
<rule provider='Fone Bro' id='182437592347598' class='esxi'>
<patterns>
<pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
@STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
@ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>
You have not included a complete syslong-ng source line for me to see what you are trying to match against so I can
not tell if you pattern will actually match the lines that you are trying to match.
At my organization we run ESX as well, and none of our lines would match the pattern that you have, but
your environment might be different.
I hope this was more clear.
Evan.
________________________________________
From: 不坏阿峰 [onlydebian at gmail.com<mailto:onlydebian at gmail.com>]
Sent: Sunday, April 28, 2013 8:24 AM
To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>; Evan Rempel
Subject: Re:Can not get DBParse match macro result (syslog-ng 3.13 debian squeeze)
thanks to your reply. i do not understand how to do now. it puzzle and trouble me some days. i read the balabit syslog-ng OSE guide documents and only have simple information in there.
how to do on this
----->>>>
If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work.
2013/4/28 <syslog-ng-request at lists.balabit.hu<mailto:syslog-ng-request at lists.balabit.hu><mailto:syslog-ng-request at lists.balabit.hu<mailto:syslog-ng-request at lists.balabit.hu>>>
Send syslog-ng mailing list submissions to
syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu><mailto:syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.balabit.hu/mailman/listinfo/syslog-ng
or, via email, send a message with subject or body 'help' to
syslog-ng-request at lists.balabit.hu<mailto:syslog-ng-request at lists.balabit.hu><mailto:syslog-ng-request at lists.balabit.hu<mailto:syslog-ng-request at lists.balabit.hu>>
You can reach the person managing the list at
syslog-ng-owner at lists.balabit.hu<mailto:syslog-ng-owner at lists.balabit.hu><mailto:syslog-ng-owner at lists.balabit.hu<mailto:syslog-ng-owner at lists.balabit.hu>>
When replying, please edit your Subject line so it is more specific
than "Re: Contents of syslog-ng digest..."
Today's Topics:
1. Can not get DBParse match macro result (syslog-ng 3.13
debian squeeze) (????)
2. Re: Can not get DBParse match macro result (syslog-ng 3.13
debian squeeze) (Evan Rempel)
----------------------------------------------------------------------
Message: 1
Date: Sat, 27 Apr 2013 22:34:50 +0800
From: ???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com><mailto:onlydebian at gmail.com<mailto:onlydebian at gmail.com>>>
Subject: [syslog-ng] Can not get DBParse match macro result (syslog-ng
3.13 debian squeeze)
To: syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu><mailto:syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
Message-ID:
<CA+SSH2oBB2-WWvQksbchVVoyhfZbdVvDR=V7wJ1EJdvE6Zx9zg at mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg at mail.gmail.com><mailto:V7wJ1EJdvE6Zx9zg at mail.gmail.com<mailto:V7wJ1EJdvE6Zx9zg at mail.gmail.com>>>
Content-Type: text/plain; charset="iso-8859-1"
when use pdbtool do match test, it is success. but from syslog-ng can not
return result of macro
i can not get macro result. for example, ${.esxi.month} no value, same
as ${.esxi.host_ip} ${.esxi.time}
test log output ,just like this.
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
do the pdbtool test, it's ok. wish someone can give me some solution and
help. i have search some mail list but i can not get the right solution.
thanks a lot.
root at debian:~# pdbtool match -D -c -p
/etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31
192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd
0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path
"vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Act:NONE"
Pattern matching part:
@STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@
@STRING:.esxi.time=15:17:31@@IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program=
vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
(0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0.
Act:NONE@@ANYSTRING:.esxi.message=cpu11:8203)NMP:
nmp_ThrottleLogForDevice:2319: Cmd
0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
Matching part:
Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
sense data: 0x5 0x20 0x0. Act:NONE
Values:
MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP:
nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev
mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid
sense data: 0x5 0x20 0x0. Act:NONE
PROGRAM=ESXI
.classifier.class=esxi
.classifier.rule_id=182437592347598
.esxi.month=Apr
.esxi.date=26
.esxi.time=15:17:31
.esxi.host_ip=192.168.88.71
.esxi.program= vmkernel
.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a
(0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0
Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
root at debian:~#
my configuration like as below
######## esxi_pattern.xml ############
<?xml version="1.0" encoding="utf-8"?>
<patterndb version='3' pub_date='2009-04-17'>
<ruleset name='esxi' id='123456678'>
<pattern>ESXI</pattern>
<rules>
<rule provider='Fone Bro' id='182437592347598' class='esxi'>
<patterns>
<pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@
@STRING:.esxi.time::@@IPv4:.esxi.host_ip:@
@ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>
######## syslog-ng.conf ########
#####Parser#####
parser pattern_db {
db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
};
#Check pattern matching
destination udp_esxi_output {
file("/var/log/pattern_output"
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST
IP ${.esxi.host_ip},${.esxi.message}\n")
template_escape(no));
};
#####Log#####
log {
source(s_network);
parser(pattern_db);
destination(udp_esxi_output);
};
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/20e80756/attachment.html
------------------------------
Message: 2
Date: Sat, 27 Apr 2013 16:10:02 +0000
From: Evan Rempel <erempel at uvic.ca<mailto:erempel at uvic.ca><mailto:erempel at uvic.ca<mailto:erempel at uvic.ca>>>
Subject: Re: [syslog-ng] Can not get DBParse match macro result
(syslog-ng 3.13 debian squeeze)
To: "Syslog-ng users' and developers' mailing list"
<syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu><mailto:syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>>
Message-ID: <q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com<mailto:q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com><mailto:q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com<mailto:q8vb966l4qe0219lsusm5ju4.1367078999747 at email.android.com>>>
Content-Type: text/plain; charset="iso-2022-jp"
It would appear that you have everything correct when the "PROGRAM" is ESXI but the log line as syslog-ng sees it has a PROGRAM of "system" according to your test log output.
If you change the patterndb ruleset pattern to use a program of system rather than ESXI I think it would work.
Evan Rempel 250.271.7691<tel:250.271.7691><tel:250.271.7691<tel:250.271.7691>>
University Systems, University of Victoria
???? <onlydebian at gmail.com<mailto:onlydebian at gmail.com><mailto:onlydebian at gmail.com<mailto:onlydebian at gmail.com>>> wrote:
when use pdbtool do match test, it is success. but from syslog-ng can not return result of macro
i can not get macro result. for example, ${.esxi.month} no value, same as ${.esxi.host_ip} ${.esxi.time}
test log output ,just like this.
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
=== system,error,critical, HOST IP ,
do the pdbtool test, it's ok. wish someone can give me some solution and help. i have search some mail list but i can not get the right solution. thanks a lot.
root at debian:~# pdbtool match -D -c -p /etc/syslog-ng/patterndb/esxi_pattern.xml -P ESXI -M "Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev "mpx.vmhba0:C0:T0:L0" on path "vmhba0:C0:T0:L0" Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE"
Pattern matching part:
@STRING:.esxi.month=Apr@ @STRING:.esxi.date=26@ @STRING:.esxi.time=15:17:31@ @IPv4:.esxi.host_ip=192.168.88.71@@ESTRING:.esxi.program= vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@ @ANYSTRING:.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE@
Matching part:
Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
Values:
MESSAGE=Apr 26 15:17:31 192.168.88.71 vmkernel: cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
PROGRAM=ESXI
.classifier.class=esxi
.classifier.rule_id=182437592347598
.esxi.month=Apr
.esxi.date=26
.esxi.time=15:17:31
.esxi.host_ip=192.168.88.71
.esxi.program= vmkernel
.esxi.message=cpu11:8203)NMP: nmp_ThrottleLogForDevice:2319: Cmd 0x1a (0x4124444a6280, 0) to dev mpx.vmhba0:C0:T0:L0 on path vmhba0:C0:T0:L0 Failed: H:0x0 D:0x2 P:0x0 Valid sense data: 0x5 0x20 0x0. Act:NONE
root at debian:~#
my configuration like as below
######## esxi_pattern.xml ############
<?xml version="1.0" encoding="utf-8"?>
<patterndb version='3' pub_date='2009-04-17'>
<ruleset name='esxi' id='123456678'>
<pattern>ESXI</pattern>
<rules>
<rule provider='Fone Bro' id='182437592347598' class='esxi'>
<patterns>
<pattern>@STRING:.esxi.month:@ @STRING:.esxi.date:@ @STRING:.esxi.time::@ @IPv4:.esxi.host_ip:@@ESTRING:.esxi.program::@ @ANYSTRING:.esxi.message@</pattern>
</patterns>
</rule>
</rules>
</ruleset>
</patterndb>
######## syslog-ng.conf ########
#####Parser#####
parser pattern_db {
db_parser( file("/etc/syslog-ng/patterndb/esxi_pattern.xml"));
};
#Check pattern matching
destination udp_esxi_output {
file("/var/log/pattern_output"
template("=== $PROGRAM,${.esxi_month} ${.esxi.date} ${.esxi.time} HOST IP ${.esxi.host_ip},${.esxi.message}\n")
template_escape(no));
};
#####Log#####
log {
source(s_network);
parser(pattern_db);
destination(udp_esxi_output);
};
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130427/2f67c039/attachment-0001.htm
------------------------------
_______________________________________________
syslog-ng maillist - syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu><mailto:syslog-ng at lists.balabit.hu<mailto:syslog-ng at lists.balabit.hu>>
https://lists.balabit.hu/mailman/listinfo/syslog-ng
End of syslog-ng Digest, Vol 96, Issue 25
*****************************************
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20130428/d0b3a38c/attachment-0001.htm
More information about the syslog-ng
mailing list