[syslog-ng] pattern usage and optimization

[Joel Carnat]

Thank you very much for this detailed explanation.
This makes it very clear now. I'll write my patterns the "syslog-ng way" 
:)

Le 01/10/2018 10:20, Fabien Wernli a écrit :
> Hi Joel,
> 
> The inner workings of patterndb and grok are very different, so you 
> can't
> really use them the same way. One of the consequences is as you've 
> already
> discovered that you sometimes need two instead of one pattern.
> 
> This might seem a limitation when moving from another tool, but is has
> reasons and one of the advantages you'll see over time with patterndb 
> are
> its speed: it's really fast. Also you get unit tests (example messages) 
> and
> you can embed any template function into the rules for instance to 
> munge or
> enrich the data.
> 
> Here are a few rules that apply to your example:
> 
> 1. Don't use patterns at the start, as these will mess up the radix 
> tree:
> 
>    > @ESTRING:EVENT: from @user @ESTRING:USERNAME: @@ESTRING:IP: @port
> 
>    Use literals instead:
> 
> 	 | Disconnected from user @ESTRING:USERNAME: @@ESTRING:IP: @port
> 
> 2. There is no regexp like grouping, so you can't say A or B or C. 
> There
>    *is* the @PCRE@ parser, but it doesn't allow to extract the matched 
> value
>    You've got two options here:
> 
>    a. Use multiple patterns:
> 
> 	 | Disconnected from user @ESTRING:USERNAME: @
> 	 | Disconnected from invalid user @ESTRING:USERNAME: @
> 	 | Disconnected from authenticating user @ESTRING:USERNAME: @
> 
> 	 b. Use one pattern and do some string stitching:
> 
>    | <patterns>
>    |   <pattern>Disconnected from @ESTRING:METHOD:user @@ESTRING:USER:
> @@ESTRING:IP: @port @NUMBER:PORT@</pattern>
>    | </patterns>
> 	 | <values>
> 	 |   <value name='METHOD'>$(strip "${METHOD}")</value>
>    | </values>
> 
>    The 'strip' is necessary as the pattern will catch the extra space.
> 	 Admittedly method b. is probably less readable, but if you care about
> 	 deduplication you might favour it over b.
> 
> 3. There is unfortunately no optional parser, so if you want to match 
> two
>    identical messages except for the ending, you need to use two 
> patterns if
>    you want to extract EXTRA
> 
> Cheers
> 
> 
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: 
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq