[syslog-ng] can kv-parser options be from macros

[Evan Rempel]

At my site we use the patterndb to match nearly every log message we get. We have over 9,000 patterns.

In some cases, such as the logs from iptables, there are many variations of the log line. Some are simply a different order of kv pairs, while others have one extra kv pair.

In other cases, there is a prefix text message followed by a json object. The json object may have a different order or different items.

There are other cases as well and they are all awkward to deal with.

What I already have working is using the patterndb to parse out a section of the log message into a json_data macro. Since this has been done with the patterndb I can arbitrarily set a few other things. I set a tag of parse_json

In the syslog-ng configuration it looks something like this:

parser p_json {
        json-parser(
                prefix(".json.")
                template("$json_data)
        );
};

# and within the log area

parser(pattern_db_patterns);
If ( filter( f_parse_json ) ) {
   parser(p_json)
};


This results in all of the json structure to be available for other syslog-ng logic/controls/routing and forwarding to a SIEM or log visualization framework.

This works well because the json parser is always parsing json data. No controls are needed to assist with the parsing.

When I look at the kv parser (and other parsers), the kv data comes in many different formats. Different pair separators and different value separators.

What I would like to do is use the patterndb to parse a message to extract a kv pairs section into a kv_data macro. Sine the patterndb is specific to the one type of message, it is also known at that time which pair-separator and the value-separator values should be used to parse the kv pairs. I would like to set  kv_vsep an a kv_psep macros in the patterndb for this message.

Then my syslog-ng configuration would look something like:

parser p_kv {
        kv-parser(
                prefix(".kv.")
                template("$kv_data")
                pair-separator("$kv_psep:- ")
                value-separator("$kv_vsep:-=")
        );
};

# and in the log area

if ( filter(f_parse_kv) ) {
    parser(p_kv);
};


This would allow the patternd to identify any type of kv_pair data AND pass the needed parsing details through to the kv_parser.

Without the ability to pass these value to the kv_parser I have to pre-determine all of the combinations of pair-separator and value-separator and make uniquely named kv parsers for each combination. Then I would need to pass from the patterndb a parser name rather than the parser options. Finally I would have to have a section of

if ( filter(f_parse_kv_name1 ) {
    parser(p_kv_name1);
};

Fore EVERY combination of pair-separator and value-separator. It can be done, but the configs need to be updated for a change it data, which just seems poor and it requires more if evaluations at runtime for each message.

Along with this, if both the pair-separator AND the value-separator are specified, this git issue should be addressed as well

https://github.com/syslog-ng/syslog-ng/issues/3953


Thanks for listening.

--------------
P.S. Further use case for those interested

We actually assign two additional macros in the patterndb is some cases.

  1.
Tag of re_match
  2.
Value of re_match_message

The re_match_message is any combination of static text and macros parsed from the original message. The intent is to be able to apply our patterndb to a value from one of the parsers (json, kv, cvs, column)

Our parsing is 4 phase

  1.
patterndb parse the original message pulling out the json data to parse. The re_match_message is incorrectly populated at this time since the json values are not yet parsed
  2.
Json parse the parsed json object from 1 (json_data) - or any other specialized parser.
  3.
patterndb with the same message as #1 just for the purpose of correctly populating the re_match_message
  4.
patterndb with the same PROGRAM but using re_match_message rather than MESSAGE

That final parsing will detect the real symantics of the log message based on data from within the json, kv-pair data. Is it a failure, success, login or logout. Does it warrent an incident to alert someone.
Real CEE nomenclature can be assinged to the meaning of the message. Lots of other detailed goodness specific to a message within the json, kv-pair, csv, column parsed data.

I know. I might need a faster server 🙂

--
Evan

________________________________
From: syslog-ng <syslog-ng-bounces at lists.balabit.hu> on behalf of Balazs Scheidler <bazsi77 at gmail.com>
Sent: November 2, 2024 4:17 AM
To: Syslog-ng users' and developers' mailing list <syslog-ng at lists.balabit.hu>
Subject: Re: [syslog-ng] can kv-parser options be from macros

 Not at the moment. But if you can explain what you'd like to accomplish I can help. Is it LEEF maybe?

Balázs

On Fri, Nov 1, 2024, 18:02 Evan Rempel <erempel at uvic.ca<mailto:erempel at uvic.ca>> wrote:
Does anyone know if I can use the enriching data to set macros  that can then be used as kv-parser options.
These values might change on a message by message basis.
For example:

parser p_kv {
    kv-parser (
        prefix("$kv_prefix")
        value-separator("$kv_vseparator")
        pair-separator("$kv_pseparator")
    );
};

--
Evan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20241102/f47d1755/attachment-0001.htm>