I was giving a lot of thought recently to the topic of naming name-value pairs in syslog-ng. Until now the only <a href="http://www.balabit.com/dl/html/syslog-ng-pe-v3.2-guide-admin-en.html/configuring_parsers.html">documented rule</a> is stating somewhat vaguely that whenever you use a parser you should choose a name that has at least one dot in it, and this dot must not be the initial character. This means that names like MSG or .SDATA.meta.sequenceId are reserved for syslog-ng, and APACHE.CLIENT_IP is reserved for users.<br /><br />However things became more complex with syslog-ng OSE 3.2. Let's see what sources generate name-value pairs:<br /><ul><li>traditional macros (e.g. $DATE); these are not name-value pairs per-se, but behave much like them, except that they are read-only</li><li>syslog message fields (e.g. $MSG) if the message is coming from a syslog source</li><li>filters whenever the 'store-matches' flag is set and the regexp contains groups</li><li>rewrite rules, whenever the rewrite rule specifies a thus far unknown name-value pair, e.g. set("something" value("name-value.pair"));</li><li>and of course parsers when you tell syslog-ng to parse an input as a CSV, or use db-parser together with the patterns produced by the <a href="http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=README.txt;hb=HEAD">patterndb project</a></li></ul>The latest stuff generating name-value pairs is the support for <a href="http://bazsi.blogs.balabit.com/2010/07/syslog-ng-and-process-accounting.html">process accounting logs</a>, in this case even the syslog related fields are missing and only things like "pacct.ac_comm" (to contain the program name) are defined.<br /><br />So I was thinking whether it should be "pacct.ac_comm" or ".pacct.ac_comm". With the quoted rule it should be simple: it is generated by syslog-ng itself, thus it should be in the syslog-ng namespace and should start with a dot. However in the era of <a href="http://bazsi.blogs.balabit.com/2010/07/syslog-ng-contributions-redefined.html">syslog-ng plugins</a>, what consists of syslog-ng at all?<br /><br />First, I wanted to use "pacct.ac_comm" (e.g. without a dot), because I liked this name better. I was trying to explain myself why it would not violate the rule above. The explanation I had for myself was: I'm going to "register" names such as this in the patterndb <a href="http://git.balabit.hu/?p=bazsi/syslog-ng-patterndb.git;a=blob;f=SCHEMAS.txt;hb=HEAD">SCHEMAS.txt file</a>. With this - not yet published - explanation, I've committed a patch to convert the <a href="http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=00cc47a0d34fcabc1e38e26882866f486fa7eb9e">pacctformat plugin</a> to use a dotless prefix.<br /><br />Next, I was figuring that it is true that process accounting creates name-value pairs without going through patternization, but I've felt, that nothing ensures that these name-value pairs would be directly usable, when trying to analyse the logs. The patterndb concept uses tags and schemas to convert the incoming unstructured data into a consistent structure. However, pacct may not completely match what the user needs. And, in the future, when SNMP traps or SQL table polling are going to be supported, it is going to be even more true: these name-value pairs may need a conversion: from the SNMP/pacct structure to the patterndb schema described structure in order to handle these message sources consistently with regular syslog (and to make it easy to correllate these).<br /><br />So at the end, I've <a href="http://git.balabit.hu/?p=bazsi/syslog-ng-3.2.git;a=commit;h=00cc47a0d34fcabc1e38e26882866f486fa7eb9e">committed another patch</a>, this time going back to ".pacct" as a prefix and leaving the original naming rule intact. The "pacct" prefix is up to the users to use, they may want the same information in a "pacct" schema, but that may come from data not directly tied from process accounting (e.g. from syslog messages).<br /><br />So this post is about doing nothing with regards to the naming policy, but I thought it'd be important to shed a light behind the scenes. Giving such decisions enough thought and coming up a with a long-term plan makes our lives much easier in the future.<br /><br />This post may be a bit more involved than the others, but feel free to ask me to elaborate, if you are interested.
<br><br>--<br>
Posted By Bazsi to <a href="http://bazsi.blogs.balabit.com/2010/08/syslog-ng-name-value-pair-naming.html">Bazsi's blog</a> at 8/06/2010 08:26:00 PM