[syslog-ng] getting "Error processing log message" for checkpoint logs

[Balazs Scheidler]

Hi,

although the entire sample is not visible in your email, we have a
checkpoint parser that fixes rfc5424 logs from checkpoint as it is using an
incorrect format. More specifically the SDATA field is using colons instead
of equal signs as in this example:

 <134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update";
flags:"150784"; ifdir:"inbound"; logid:"160571424";
loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}"; origin:"192.168.32.91";
originsicname:"CN=GW91,O=Domain2_Server..cuggd3"; sequencenum:"1";
time:"1521645925"; version:"5"; auth_method:"Machine Authentication (Active
Directory)"; auth_status:"Successful Login"; authentication_trial:"this is
a reauthentication for session 9a026bba"; client_name:"Active Directory
Query"; client_version:"R80.10"; domain_name:"spec.mgmt";
endpoint_ip:"192.168.32.69"; identity_src:"AD Query";
identity_type:"machine"; product:"Identity Awareness"; snid:"9a026bba";
src:"192.168.32.69"; src_machine_group:"All Machines";
src_machine_name:"yonatanad";]

we have a checkpoint-parser() component that parses and fixes this into
name-value pairs since 3.21, but with fixes in 3.26:

https://github.com/syslog-ng/syslog-ng/blob/master/scl/checkpoint/plugin.conf

To use that parser you will need to receive logs with flags(no-parse) and
then apply the checkpoint-parser() as defined above.

Also, there's a component in syslog-ng that receives messages on all sane
channels and parses/fixes them automatically, this is the
"default-network-drivers()" source, which apart from receiving on
tcp/udp/ssl + rfc5424 will properly parse cisco/checkpoint/etc logs by
applying application specific filtering/parsing rules, called application
adapters. For instance the adapter for checkpoint looks like this:

application checkpoint[syslog-raw] {
    filter {
        # "syslog" format
        message("^(<[0-9]{1,3}>)1 .* CheckPoint ") or
        # "splunk" format
        message('^time=[0-9]+\|hostname=[a-zA-Z0-9-]+\|product=Firewall');
    };
    parser { checkpoint-parser(); };
};

This tells syslog-ng that it should look for checkpoint messages in the
"syslog-raw" topic (e.g. stream of messages), apply checkpoint-parser() to
log messages where the filter above matches. As you can see it will
identify both "syslog" and "splunk" style messages emitted from checkpoint.

On Mon, May 18, 2020 at 4:15 AM Russell Fulton <r.fulton at auckland.ac.nz>
wrote:

> I am trying to ingest logs from a checkpoint firewall over a tcp
> connection:
>
> source s_checkpoint {
>     network( transport("tcp") flags(no-multi-line) port(2514)
> keep-alive(yes) flags(syslog-protocol) );
> };
>
> # added flags(syslog-protocol) on suggest of checkpoint docs for syslog-ng
>
> destination d_debug {
>             file( "/data/russell//test.log"   );
>          };
>
>
> log {
>     source( s_checkpoint );
>     destination( d_debug );
> };
>
>
> This is what turns up in the output:
>
> May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log message:
> <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -…..
>
> Any ideas on what is going on or how to find out?
>
> Russell
>
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>

-- 
Bazsi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200518/491149be/attachment.html>