[syslog-ng] quote characters (or other specials) in patterens?

19 Dec 2013

      Hello,

How should I handle quote characters (or other specials) in patterens?

(To help with redering the examples below can be found here:
https://gist.github.com/linickx/8038784)

This works and will validate as expected (working.xml)...

<patterndb version='4' pub_date='2013-12-17'>
    <ruleset name='bluecoat' id='dd001'>
        <pattern>bluecoat</pattern>
            <rules>
                <rule provider='linickx' id='nbdd001' class='system'>
                    <patterns>

<pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@
@NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - -
@ESTRING:BC_ACTION: @</pattern>
                    </patterns>
                    <examples>
                        <example>
                            <test_message program="bluecoat">10:57:56
43 10.8.26.200 - - - OBSERVED "Web Ads/Analytics"
http://googleads.g.doubleclick.net/mads/  200 TCP_CLIENT_REFRESH GET
image/png http pagead2.googlesyndication.com 80 /
pagead/images/nessie_icon_chevron_white.png - png "Mozilla/5.0 (Linux;
U; Android 4.0.4; en-gb; P76a(K3G5) Build/IMM76D) AppleWebKit/534.30
(KHTML, like Gecko) Version/4.0 Safari/534.30 (Mobile;
afma-sdk-a-v6.2.1)" 10.8.24.5 724 1277 -</t
est_message>
                            <test_values>
                                <test_value name="BC_HOUR">10</test_value>
                                <test_value name="BC_MIN">57</test_value>
                                <test_value name="BC_SEC">56</test_value>
                                <test_value name="BC_TIME_TAKEN">43</test_value>
                                <test_value
name="BC_CLIENT_ADDRESS">10.8.26.200</test_value>
                                <test_value
name="BC_ACTION">OBSERVED</test_value>
                            </test_values>
                       </example>
                    </examples>
                </rule>
            </rules>
    </ruleset>
</patterndb>

Simply by updating <pattern> with a quote for my next match (broken1.xml) ...

<pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@
@NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - -
@ESTRING:BC_ACTION: @ "</pattern>

... the whole thing bjorks (broken1_output.txt)...

[nick@localhost ~]$ pdbtool test --validate nick.xml
nick.xml validates
Key contains '@' without escaping; key='@"', value='nbdd001'
Testing message program='bluecoat' message='10:57:56 43 10.8.26.200 -
- - OBSERVED "Web Ads/Analytics"
http://googleads.g.doubleclick.net/mads/  200 TCP_CLIENT_REFRESH GET
image/png http pagead2.googlesyndication.com 80
/pagead/images/nessie_icon_chevron_white.png - png "Mozilla/5.0
(Linux; U; Android 4.0.4; en-gb; P76a(K3G5) Build/IMM76D)
AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Safari/534.30
(Mobile; afma-sdk-a-v6.2.1)" 10.8.24.5 724 1277 -'
 Wrong match name='.classifier.rule_id', value='', expected='nbdd001'
 Wrong match name='BC_HOUR', value='', expected='10'
 Wrong match name='BC_MIN', value='', expected='57'
 Wrong match name='BC_SEC', value='', expected='56'
 Wrong match name='BC_TIME_TAKEN', value='', expected='43'
 Wrong match name='BC_CLIENT_ADDRESS', value='', expected='10.8.26.200'
 Wrong match name='BC_ACTION', value='', expected='OBSERVED'
[nick@localhost ~]$

I have tried various escape methods...

(escape_traditional.xml)
<pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@
@NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - -
@ESTRING:BC_ACTION: @ \"</pattern>

or

(escape_at.xml)
<pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@
@NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - -
@ESTRING:BC_ACTION: @ @"</pattern>

I have tried various match methods...

(test_estring_stopquote.xml)
<pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@
@NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - -
@ESTRING:BC_ACTION: @ "@ESTRING:BC_CATEGORY:"@</pattern>
<test_value name="BC_CATEGORY">Web Ads/Analytics</test_value>

or

(test_estring_incquotes.xml)
<pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@
@NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - -
@ESTRING:BC_ACTION: @ @ESTRING:BC_CATEGORY: @</pattern>
<test_value name="BC_CATEGORY">"Web Ads/Analytics"</test_value>

or

(test_qstring.xml)
<pattern>@NUMBER:BC_HOUR:@:@NUMBER:BC_MIN:@:@NUMBER:BC_SEC:@
@NUMBER:BC_TIME_TAKEN:@ @IPv4:BC_CLIENT_ADDRESS:@ - - -
@ESTRING:BC_ACTION: @ @QSTRING:BC_CATEGORY:"@</pattern>
<test_value name="BC_CATEGORY">Web Ads/Analytics</test_value>

But no joy! (same error output as above) Any pointers would be appreciated!

Testing carried out on:
...
Fedora release 19 (Schrödinger’s Cat)
syslog-ng-3.4.6-1.fc19.i686
Thanks in Advance,
Nick