[syslog-ng] Problems with custom patterndb

Clements, Frank fclements at corp.ptd.net
Tue Nov 29 11:56:30 CET 2011 

Evan, thank you for the tips!  I cleaned up the carriage returns and it works perfectly!  Thanks for your help, very much appreciated.

________________________________________
From: syslog-ng-bounces at lists.balabit.hu [syslog-ng-bounces at lists.balabit.hu] On Behalf Of Evan Rempel [erempel at uvic.ca]
Sent: Tuesday, November 29, 2011 1:08 AM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] Problems with custom patterndb

Please keep in mind that everything (I meand *everything* between the <patter> and </patter> is taken as a literal.
This means that your pattern starts with a carriage return, followed by a bunch of spaces and the @IPvANY...

I don't think that is what you intended

               <pattern>
                  @IPvANY:.dict.insideAddr@:@NUMBER:.dict.insidePort@ -&gt; @IPvANY:.dict.outsideAddr@:@NUMBER:.dict.outsidePort@ to @IPvANY:.dict.destAddr@:@NUMBER:.dict.destPort@
               </pattern>

try
               <pattern>@IPvANY:.dict.insideAddr@:@NUMBER:.dict.insidePort@ -&gt; @IPvANY:.dict.outsideAddr@:@NUMBER:.dict.outsidePort@ to @IPvANY:.dict.destAddr@:@NUMBER:.dict.destPort@</pattern>

let me know how that goes.

________________________________________
From: syslog-ng-bounces at lists.balabit.hu [syslog-ng-bounces at lists.balabit.hu] On Behalf Of Clements, Frank [fclements at corp.ptd.net]
Sent: Monday, November 28, 2011 8:12 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] Problems with custom patterndb

Thanks for the help.

I checked over the configuration, top-to-bottom, and there's no sign of the "no_parse" flag.  Still no luck though.  I read in the admin guide about triggered messages having to use the inject_mode(internal), I'm not sure if this applies here - but I'm not specifying anything.  My ultimate goal is correlate events together into a single message which will go to program().  As of now though, I can't even get patterndb messages to get logged to flat files.


Here's more detail:

Syslog-NG version 3.3
Using loggen to generate the logs with an input file as the source of the messages:
  /opt/syslog-ng/bin/loggen -i -D -R ~fclements/syslog.messages  --skip-tokens=1 192.168.1.10 515


The input file contains:
 NAT-UDP-C: 192.168.107.132:12260 -> 207.44.101.104:12260 to 48.70.67.223:940
 UC: c0a80b1b:1a->c0a8650b:5c40 to c0a8650c:3f

I have the following patterns created (from patterndb.xml):
         <rule id='18795aa7-eb96-454d-b954-646c0b679d44' class='system' provider='testing-provider'>
            <patterns>
               <pattern>
                  @IPvANY:.dict.insideAddr@:@NUMBER:.dict.insidePort@ -&gt; @IPvANY:.dict.outsideAddr@:@NUMBER:.dict.outsidePort@ to @IPvANY:.dict.destAddr@:@NUMBER:.dict.destPort@
               </pattern>
               <pattern>
                  @STRING:.dict.insideAddr@:@STRING:.dict.insidePort at -&gt;@STRING:.dict.outsideAddr@:@ESTRING:.dict.outsidePort: @to @STRING:.dict.destAddr@:@ESTRING:.dict.destPort: @
               </pattern>
            </patterns>
            <values></values>
         </rule>

syslog-ng.conf has the following:
options {
   threaded(yes);
   use_dns(no);
};

parser pattern_db {
   db_parser( file("/var/lib/syslog-ng/patterndb.xml") );
};

source s_remote {
        udp( ip(0.0.0.0) port(515) flags("store-legacy-msghdr"));
};
destination d_message_class {
        file( "/var/log/parsed/messages"
           template( "${.classifier.rule_id}\t${S_UNIXTIME}\t${HOST}\t${PROGRAM}\t${.dict.insideAddr}\t${.dict.insidePort}\n" )
           create_dirs(yes)
        );
};

log {
        source(s_remote);
        parser(pattern_db);
        destination( d_message_class );
};

In my template I'm using .classifier.rule_id, .dict.insideAddr and .dict.insidePort.  They all turn up blank in the file - I'm not even getting a hit on the rule_id:
        1322539328      192.168.1.10    NAT-UDP-C
        1322539328      192.168.1.10    UC

-Frank

________________________________________
From: syslog-ng-bounces at lists.balabit.hu [syslog-ng-bounces at lists.balabit.hu] On Behalf Of Evan Rempel [erempel at uvic.ca]
Sent: Monday, November 28, 2011 5:28 PM
To: Syslog-ng users' and developers' mailing list
Subject: Re: [syslog-ng] Problems with custom patterndb

What you have is correct (I tested it on my parser database)
given that the message getting passed to the parser is

192.168.107.132:12260 -> 207.44.101.104:12260 to 48.70.67.223:940

So it really depends if this message is coming from real syslog source
or if it is coming from a "no_parse" source.

Evan.

Clements, Frank wrote:
> Hello Everyone!
>
> I've been trying to get some custom patterns put together to do some log
> correlation and I'm having one hell of a time getting a working
> pattern.  I think I need a second, third, forth set of eyes on this ...
> Any help is appreciated!
>
> Message: Nov 25 12:02:27 GENERATED NAT-UDP-C: 192.168.107.132:12260 ->
> 207.44.101.104:12260 to 48.70.67.223:940
> Pattern: @IPvANY:.dict.insideAddr@:@NUMBER:.dict.insidePort@ -&gt;
> @IPvANY:.dict.outsideAddr@:@NUMBER:.dict.outsidePort@ to
> @IPvANY:.dict.destAddr@:@NUMBER:.dict.destPort@
>
> I've looked at a few examples from the community patterns, but nothing
> in this stands out as being "wrong".
>
> Thanks
>
> -
> Frank W Clements
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>


--
Evan Rempel                               erempel at uvic.ca
Senior Systems Administrator                 250.721.7691
Unix Services, University Systems, University of Victoria
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

More information about the syslog-ng mailing list