<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"></head><body ><div>Thanks. Seems obvious now. </div><div>The pdbtool test *was* parsing but syslog-ng was not...</div><div>Again *doh* :-)</div><div><br></div><div><br></div><div><br></div><div><div style="font-size:75%;color:#575757">Sent from my Verizon Wireless 4G LTE Smartphone</div></div> <br><br><br>-------- Original message --------<br>From: Evan Rempel <erempel@uvic.ca> <br>Date: 01/10/2014 2:03 PM (GMT-05:00) <br>To: syslog-ng@lists.balabit.hu <br>Subject: Re: [syslog-ng] how to use macros from patterndb in syslog-ng.conf <br> <br><br>With no-parse you will not get a program name, which will prevent the pattern from matching in patterndb<br><br>Evan.<br><br>On 01/10/2014 10:59 AM, jrhendri@roadrunner.com wrote:<br>> DOH!!!<br>> quoth the bard Homer Simpson...<br>><br>> in my testing, I was appending syslog messages to "testfile" like this:<br>> head -10 user.info.2014.01.08 > testfile<br>><br>> and using "flags(no-parse)" on that file source which looks like this:<br>><br>> [n0142566@VDDP13E-0D6B677 ~]$ cat testfile<br>> Jan 8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN<br>> Jan 8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN<br>> [n0142566@VDDP13E-0D6B677 ~]$<br>><br>><br>><br>> not quite sure how / why - but that was breaking this even though this worked:<br>><br>> [n0142566@VDDP13E-0D6B677 ~]$ pdbtool match -f testfile -p ./log/juniper_db.xml<br>> HOST=10.192.225.12<br>> MESSAGE=RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN<br>> PROGRAM=RT_FLOW<br>> LEGACY_MSGHDR=RT_FLOW:<br>> .classifier.class=system<br>> .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d<br>> J.MSG.SRC=146.150.19.13<br>> J.MSG.SPORT=46188<br>> J.MSG.DST=10.178.128.38<br>> J.MSG.DPORT=11000<br>> TAGS=.classifier.system<br>><br>> HOST=10.192.225.12<br>> MESSAGE=RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN<br>> PROGRAM=RT_FLOW<br>> LEGACY_MSGHDR=RT_FLOW:<br>> .classifier.class=system<br>> .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d<br>> J.MSG.CLOSE.REASON=TCP RST<br>> J.MSG.SRC=10.178.200.127<br>> J.MSG.SPORT=43833<br>> J.MSG.DST=10.181.72.46<br>> J.MSG.DPORT=2059<br>> TAGS=.classifier.system<br>><br>><br>> After I took out flags(no-parse) on the file source, it works as (I) expected ...<br>><br>> I am still curious what I overlooked - misunderstood. But it appears to be related to how the messages were parsed on the source end.<br>><br>> Thanks again!<br>> Jim<br>><br>><br>><br>><br>> ---- jrhendri@roadrunner.com wrote:<br>>> this seems like a simple thing to do, but a few days of searching and some hours of testing has not shown me the answer.<br>>><br>>> Essentially I want to parse log events and output select fields, so I have created a basic patterndb xml file.<br>>><br>>> <patterndb version='3' pub_date='2014-01-09'><br>>> <ruleset name='patternize' id='6cb77f11-6c9b-ee4e-9f62-b97224d4384c'><br>>> <rules><br>>> <rule id='2a270520-2ff7-7048-a088-b03d5b3b5f7d' class='system' provider='patternize'><br>>> <!-- support: 1 --><br>>> <patterns><br>>> <pattern>RT_FLOW_SESSION_CLOSE: session closed @ESTRING:J.MSG.CLOSE.REASON::@ @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@-&gt;@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern><br>>> <pattern>RT_FLOW_SESSION_CREATE: session created @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT@-&gt;@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern><br>>> </patterns><br>>> </rule><br>>> </rules><br>>> </ruleset><br>>> </patterndb><br>>><br>>> This tests fine using pdbtool on sample data in a file. (cut from a real syslog file of logs)<br>>> Inside syslog-ng.conf I want to use the parsed values as a template:<br>>> source s_testfile {<br>>> file("/home/n0142566/testfile"<br>>> flags(no-parse) );<br>>> };<br>>><br>>> filter f_juniper_session_create {<br>>> match("RT_FLOW_SESSION_CREATE" value( "MESSAGE" ) );<br>>> };<br>>> filter f_juniper_session_close {<br>>> match("RT_FLOW_SESSION_CLOSE" value( "MESSAGE" ) );<br>>> };<br>>><br>>> parser juniper_db {<br>>> db-parser (<br>>> file("/home/n0142566/log/juniper_db.xml")<br>>> );<br>>> };<br>>><br>>> destination d_local_create {<br>>> file("/home/n0142566/log/messages-create-$HOST"<br>>> template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}\n") );<br>>> };<br>>><br>>> destination d_local_close {<br>>> file("/home/n0142566/log/messages-close-$HOST"<br>>> #!# template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}, ${J.MSG.CLOSE.REASON}\n") );<br>>> template("${J.MSG}\n") );<br>>> };<br>>><br>>><br>>> log {<br>>> source(s_testfile);<br>>> filter(f_juniper_session_create);<br>>> parser(juniper_db);<br>>> destination(d_local_create);<br>>> };<br>>> log {<br>>> source(s_testfile);<br>>> filter(f_juniper_session_close);<br>>> parser(juniper_db);<br>>> destination(d_local_close);<br>>> };<br>>><br>>><br>>> But when I run syslog-ng and append events to "testfile" the output is simply comma separated blanks :-(<br>>><br>>> I am quite sure I am missing something *extremely* basic - but at this point I thought I would ask clearer minds for help!!<br>>><br>>> Thanks,<br><br>______________________________________________________________________________<br>Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng<br>Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng<br>FAQ: http://www.balabit.com/wiki/syslog-ng-faq<br><br></body>