[syslog-ng] pattern usage and optimization

[Fabien Wernli]

Hi Joel,

The inner workings of patterndb and grok are very different, so you can't
really use them the same way. One of the consequences is as you've already
discovered that you sometimes need two instead of one pattern.

This might seem a limitation when moving from another tool, but is has
reasons and one of the advantages you'll see over time with patterndb are
its speed: it's really fast. Also you get unit tests (example messages) and
you can embed any template function into the rules for instance to munge or
enrich the data.

Here are a few rules that apply to your example:

1. Don't use patterns at the start, as these will mess up the radix tree:

   > @ESTRING:EVENT: from @user @ESTRING:USERNAME: @@ESTRING:IP: @port

   Use literals instead:

	 | Disconnected from user @ESTRING:USERNAME: @@ESTRING:IP: @port
 
2. There is no regexp like grouping, so you can't say A or B or C. There
   *is* the @PCRE@ parser, but it doesn't allow to extract the matched value
   You've got two options here:

   a. Use multiple patterns:

	 | Disconnected from user @ESTRING:USERNAME: @
	 | Disconnected from invalid user @ESTRING:USERNAME: @
	 | Disconnected from authenticating user @ESTRING:USERNAME: @

	 b. Use one pattern and do some string stitching:

   | <patterns>
   |   <pattern>Disconnected from @ESTRING:METHOD:user @@ESTRING:USER: @@ESTRING:IP: @port @NUMBER:PORT@</pattern>
   | </patterns>
	 | <values>
	 |   <value name='METHOD'>$(strip "${METHOD}")</value>
   | </values>

   The 'strip' is necessary as the pattern will catch the extra space.
	 Admittedly method b. is probably less readable, but if you care about
	 deduplication you might favour it over b.

3. There is unfortunately no optional parser, so if you want to match two
   identical messages except for the ending, you need to use two patterns if
   you want to extract EXTRA

Cheers

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 2801 bytes
Desc: not available
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20181001/12c57a63/attachment-0001.bin>