[syslog-ng] db-parser: tag and value support

ILLES, Marton illes.marton at balabit.hu
Tue Aug 18 21:27:37 CEST 2009


Hi,

I had some time recently and did some db-parser improvements in my
syslog-ng 3.1 branch. Some of the features are already integrated to 3.1
mainline by Bazsi, but I though it is worthwhile to write about them. I
have posted it to my blog (http://marci.blogs.balabit.com/), but here it
goes as well.

In one of my earlier post I wrote about adding tag support to syslog-ng.
It has been already merged to the 3.1 mainline by Bazsi. To make the
feature even more useful the version 2 patterndb had support for
specifying tags in each rule to tag the matching messages. So far this
functionality was missing, but now it is possible to tag messages
directly from the patterndb configuration so you can latter filter based
on these tags. It is more versatile than using classification as you can
specify multiple tags and tag matching is way more faster than matching
the class of the message. (Though it is not possible to use the tags in
macros as you can do with classes. eg.: you can use the
${.classifier.class} macro while you can not include the tags)

Besides tags patterndb gained support for using static value assignment
to messages. With patterndb version 3 (which is backward compatible with
version 2) it is possible to specify named values in each rule. When a
message matches a given rule all the specified values are assigned to
the message (just like names parsers like @STRING:myvalue@) and these
values can be latter used in filters or in macros. (See my previous post
on parsers.) Also it is possible to use macros in values, so you can
refer back to any parsed part of the message. It is probably most useful
if you want to specially store the messages in some structured form
where you need to fill all required values.

With these addition you can use rules like this:


<rule id="1" provider="balabit" class="system">
 <patterns>
  <pattern>my pattern @STRING:my_parsed_value@</pattern>
 </patterns>
 <tags>
  <tag>tag1</tag>
  <tag>tag2</tag>
 </tags>
 <values>
  <value name="myvalue1">foo</value>
  <value name="myvalue2">bar ${myvalue1}</value>
 </values>
</rule>


Version 3 of the patterndb has been also extended so it can hold example
messages for a given pattern. It is really handy if you need to test
your pattern automatically or simply you want to document the original
message that you created the pattern for. With the examples you can also
specify expected values that are parsed or set from the message.

The previous example can be extended this way:


<rule id="1" provider="balabit" class="system">
 <patterns>
  <pattern>my pattern @STRING:my_parsed_value@</pattern>
 </patterns>
 <tags>
  <tag>tag1</tag>
  <tag>tag2</tag>
 </tags>
 <values>
  <value name="myvalue1">foo</value>
  <value name="myvalue2">bar ${myvalue1}</value>
 </values>
 <examples>
  <test_message program="programname">my pattern test</test_message>
  <test_values>
   <test_value name="myvalue1">foo</test_value>
   <test_value name="myvalue2">bar foo</test_value>
   <test_value name="my_parser_value">test</test_value>
  </test_values>
  <test_tags>
   <test_tag>tag1</test_tag>
   <test_tag>tag2</test_tag>
  </test_tags>
  <example>
 </examples>
</rule>


You can find the source code in my 3.1 git tree at
http://git.balabit.hu/?p=marci/syslog-ng-3.1.git;a=summary

Happy tagging!

M
-- 
Key fingerprint = F78C 25CA 5F88 6FAF EA21 779D 3279 9F9E 1155 670D



More information about the syslog-ng mailing list