[syslog-ng] how to use macros from patterndb in syslog-ng.conf

[Evan Rempel]

With no-parse you will not get a program name, which will prevent the pattern from matching in patterndb

Evan.

On 01/10/2014 10:59 AM, jrhendri at roadrunner.com wrote:
> DOH!!!
> quoth the bard Homer Simpson...
>
> in my testing, I was appending syslog messages to "testfile" like this:
> head -10 user.info.2014.01.08 > testfile
>
> and using "flags(no-parse)" on that file source which looks like this:
>
> [n0142566 at VDDP13E-0D6B677 ~]$ cat testfile
> Jan  8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN
> Jan  8 20:00:00 10.192.225.12 RT_FLOW: RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN
> [n0142566 at VDDP13E-0D6B677 ~]$
>
>
>
> not quite sure how / why - but that was breaking this even though this worked:
>
> [n0142566 at VDDP13E-0D6B677 ~]$ pdbtool match -f testfile -p ./log/juniper_db.xml
> HOST=10.192.225.12
> MESSAGE=RT_FLOW_SESSION_CREATE: session created 146.150.19.13/46188->10.178.128.38/11000 None 146.150.19.13/46188->10.178.128.38/11000 None None 6 3798 Int_NonProd_CORE_Liberty Int_NonProd_CORE_Secure 180869281 N/A(N/A) reth0.3931 UNKNOWN UNKNOWN UNKNOWN
> PROGRAM=RT_FLOW
> LEGACY_MSGHDR=RT_FLOW:
> .classifier.class=system
> .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d
> J.MSG.SRC=146.150.19.13
> J.MSG.SPORT=46188
> J.MSG.DST=10.178.128.38
> J.MSG.DPORT=11000
> TAGS=.classifier.system
>
> HOST=10.192.225.12
> MESSAGE=RT_FLOW_SESSION_CLOSE: session closed TCP RST: 10.178.200.127/43833->10.181.72.46/2059 junos-tcp-any 10.178.200.127/43833->10.181.72.46/2059 None None 6 3384 Int_NonProd_CORE_Secure Int_NonProd_CORE_Liberty 180781521 2(100) 1(60) 3 UNKNOWN UNKNOWN N/A(N/A) reth0.3930 UNKNOWN
> PROGRAM=RT_FLOW
> LEGACY_MSGHDR=RT_FLOW:
> .classifier.class=system
> .classifier.rule_id=2a270520-2ff7-7048-a088-b03d5b3b5f7d
> J.MSG.CLOSE.REASON=TCP RST
> J.MSG.SRC=10.178.200.127
> J.MSG.SPORT=43833
> J.MSG.DST=10.181.72.46
> J.MSG.DPORT=2059
> TAGS=.classifier.system
>
>
> After I took out flags(no-parse) on the file source, it works as (I) expected ...
>
> I am still curious what I overlooked - misunderstood. But it appears to be related to how the messages were parsed on the source end.
>
> Thanks again!
> Jim
>
>
>
>
> ---- jrhendri at roadrunner.com wrote:
>> this seems like a simple thing to do, but a few days of searching and some hours of testing has not shown me the answer.
>>
>> Essentially I want to parse log events and output select fields, so I have created a basic patterndb xml file.
>>
>> <patterndb version='3' pub_date='2014-01-09'>
>>    <ruleset name='patternize' id='6cb77f11-6c9b-ee4e-9f62-b97224d4384c'>
>>      <rules>
>>        <rule id='2a270520-2ff7-7048-a088-b03d5b3b5f7d' class='system' provider='patternize'>
>>          <!-- support: 1 -->
>>          <patterns>
>>            <pattern>RT_FLOW_SESSION_CLOSE: session closed @ESTRING:J.MSG.CLOSE.REASON::@ @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT at -&gt;@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern>
>>            <pattern>RT_FLOW_SESSION_CREATE: session created @IPv4:J.MSG.SRC@/@NUMBER:J.MSG.SPORT at -&gt;@IPv4:J.MSG.DST@/@NUMBER:J.MSG.DPORT@ </pattern>
>>          </patterns>
>>        </rule>
>>      </rules>
>>    </ruleset>
>> </patterndb>
>>
>> This tests fine using pdbtool on sample data in a file. (cut from a real syslog file of logs)
>> Inside syslog-ng.conf I want to use the parsed values as a template:
>> source s_testfile {
>>      file("/home/n0142566/testfile"
>>       flags(no-parse) );
>> };
>>
>> filter f_juniper_session_create {
>>    match("RT_FLOW_SESSION_CREATE" value( "MESSAGE" ) );
>> };
>> filter f_juniper_session_close {
>>    match("RT_FLOW_SESSION_CLOSE" value( "MESSAGE" ) );
>> };
>>
>> parser juniper_db {
>>    db-parser (
>>      file("/home/n0142566/log/juniper_db.xml")
>>    );
>> };
>>
>> destination d_local_create {
>>          file("/home/n0142566/log/messages-create-$HOST"
>>            template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}\n") );
>> };
>>
>> destination d_local_close {
>>          file("/home/n0142566/log/messages-close-$HOST"
>> #!#          template("${J.MSG.SRC}, ${J.MSG.SPORT}, ${J.MSG.DST}, ${J.MSG.DPORT}, ${J.MSG.CLOSE.REASON}\n") );
>>            template("${J.MSG}\n") );
>> };
>>
>>
>> log {
>>    source(s_testfile);
>>    filter(f_juniper_session_create);
>>    parser(juniper_db);
>>    destination(d_local_create);
>> };
>> log {
>>    source(s_testfile);
>>    filter(f_juniper_session_close);
>>    parser(juniper_db);
>>    destination(d_local_close);
>> };
>>
>>
>> But when I run syslog-ng and append events to "testfile" the output is simply comma separated blanks :-(
>>
>> I am quite sure I am missing something *extremely* basic - but at this point I thought I would ask clearer minds for help!!
>>
>> Thanks,