[syslog-ng] Match logs with variable lengths

[Evan Rempel]

There are a number of ways to accomplish what you want.

I have not tested any of these.

PCRE

Using PCRE does not require that the entire pattern use pcre. Doing that 
will actually slow down the patterndb. This would capture the leading 
space. " [ECS 192.168.0.0/24/0]"

<pattern>client @@@ESTRING:dns_clientid: @@IPvANY:src_ip@#@NUMBER:src_port@ (@ESTRING:dns_original_query:)@: query: @ESTRING:dns_qname: @IN @ESTRING:dns_type: @@ESTRING:dns_record_flags: @(@IPvANY:dns_server@)@PCRE::( \[ECS [0-9.]+/\d+/\d+\])?@</pattern>

ANYSTRING

There is an implied @ANYSTRING@ at the end of every pattern. This is the 
reason that your current pattern matches both lines. You can capture the 
result of ANYSTRING, but would also include the leading space " [ECS 
192.168.0.0/24/0]"

<pattern>client @@@ESTRING:dns_clientid: @@IPvANY:src_ip@#@NUMBER:src_port@ (@ESTRING:dns_original_query:)@: query: @ESTRING:dns_qname: @IN @ESTRING:dns_type: @@ESTRING:dns_record_flags: @(@IPvANY:dns_server@)@ANYSTRING@</pattern>


Combination of PARSERS

I think that the SET parser can return null results, so combining it 
with ANYSTRING the ending can be captured without the leading space 
"[ECS 192.168.0.0/24/0]"

<pattern>client @@@ESTRING:dns_clientid: @@IPvANY:src_ip@#@NUMBER:src_port@ (@ESTRING:dns_original_query:)@: query: @ESTRING:dns_qname: @IN @ESTRING:dns_type: @@ESTRING:dns_record_flags: @(@IPvANY:dns_server@)@SET:: @@ANYSTRING@</pattern>


Multiple patterns (my preferred solution)

If multiple patterns match the same log line, then the first pattern in 
the patterndb is used. Abusing this behaviour, if you place the longer 
pattern first in the patterndb, then it will be used to match the longer 
log line. This method can permit you to capture individual parts of the 
extra text so is more flexible for other use cases.

<pattern>client @@@ESTRING:dns_clientid: @@IPvANY:src_ip@#@NUMBER:src_port@ (@ESTRING:dns_original_query:)@: query: @ESTRING:dns_qname: @IN @ESTRING:dns_type: @@ESTRING:dns_record_flags: @(@IPvANY:dns_server@) [@IPvANY:network@/@NUMBER:prefix@/@NUMBER@]</pattern>
<pattern>client @@@ESTRING:dns_clientid: @@IPvANY:src_ip@#@NUMBER:src_port@ (@ESTRING:dns_original_query:)@: query: @ESTRING:dns_qname: @IN @ESTRING:dns_type: @@ESTRING:dns_record_flags: @(@IPvANY:dns_server@)</pattern>

I hope this helps.

Evan

On 2022-12-06 17:54, Mik J wrote:
> Notice: This message was sent from outside the University of Victoria email system. Please be cautious with links and sensitive information.
>
>
> Hello,
>
> I'm new to syslog-ng and I'm trying to match my DNS logs.
>
> I have either
> Dec  6 01:44:49 mydnsserver named[66669]: client @0x53f33c122d0 1.2.3.4#56610 (my.original.query.com): query: my.qname.com IN A -E(0)DC (1.1.1.1)
> or
> Dec  6 01:44:49 mydnsserver named[66669]: client @0x53f33c122d0 1.2.3.4#56610 (my.original.query.com): query: my.qname.com IN A -E(0)DC (1.1.1.1) [ECS 192.168.0.0/24/0]
>
> So the field [ECS 192.168.0.0/24/0] is optional and either exist or not
>
> To match fields I use this pattern
>   <pattern>client @@@ESTRING:dns_clientid: @@IPvANY:src_ip@#@NUMBER:src_port@ (@ESTRING:dns_original_query:)@: query: @ESTRING:dns_qname: @IN @ESTRING:dns_type: @@ESTRING:dns_record_flags: @(@IPvANY:dns_server@)</pattern>
>
> and it matches both lines for the existing fields
>
> But how can I match this optional part at the end of the line: [ECS 192.168.0.0/24/0]
>
> Should I use @PCRE@ ? Or is it possible with other pattern parsers
> If yes why use other pattern parsers and not build the matching around @PCRE@ for the entire line which would make think simple to translate from grok or other tools.
>
> Thank you
>