[syslog-ng] [FORGED] Re: [FORGED] getting "Error processing log message" for checkpoint logs

Russell Fulton r.fulton at auckland.ac.nz
Tue May 19 19:10:30 UTC 2020



Russell at fulton.nz

On 19/05/2020, at 6:08 PM, Balazs Scheidler <bazsi77 at gmail.com> wrote:


Yes.

My bad!  There were two questions!


The one you missed was that when i pass the logs to elastic search i don’t get the parsed n v pairs.  I get the standard syslog record with all the details in MESSAGE

THe question about the prefix was an afterthought, one i should have squashed!

Russell

On Tue, May 19, 2020, 06:25 Russell Fulton <r.fulton at auckland.ac.nz<mailto:r.fulton at auckland.ac.nz>> wrote:
Thanks Balazs.

I am using 3.26 and have amended my config to disable the default parser and specify the new one:

source s_checkpoint {
    network( transport("tcp") flags(no-multi-line) port(2514)  flags(no-parse) );
};

log {
    source( s_checkpoint );
    parser { checkpoint-parser(); };
    destination( d_debug );
};

I no long get the parser errors and the output in the debug file looks like this:

May 19 04:07:16 smartlog01 CheckPoint[14153]: [action:"Accept"; conn_direction:"Outgoing"; flags:"6422528"; ifdir:"inbound"; ifname:"bond2.440"; logid:"321"; loguid:"{0x0,0x0,0x0,0x0}"; origin:"172.19.2.38"; originsicname:"CN=br-cpf4,O=smartcenter01.net.auckland.ac<http://smartcenter01.net.auckland.ac>.nz.qgmmde"; sequencenum:"47"; time:"1589861236"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={22CAD420-4CBB-2340-917B-1C32C546DEB1};mgmt=smartcenter01;date=1589757719;policy_name=Border_Policy\]"; connection_luuid:"01001000-8300-00c0-5ec3-5b3800000001"; hll_key:"533041188277717813"; product:"Application Control"; method:"GET"; resource:"http://storage.googleapis.com/update-delta/gkmgaooipdjhmangpemjhigmamcehddo/81.235.200/80.230.200/3d83bc1481c09741d8bd52f2cef8af3e2206f2f0b5c816b46d9725be61977a6f.crxd"; ]

so they are clearly going through the parser.

when I log it to elastic search I just get the standard fields with the body in MESSAGE, no parsed N V pairs.
destination d_elastic_checkpoint {

  elasticsearch_http(
    index("checkpoint_${YEAR}.${MONTH}.${DAY}")
   type("_doc")
   persist-name("checkpoint")
   template("$(format-json --scope nv-pairs --exclude HOST_FROM
                                            --exclude HOST
                                            --exclude SOURCE
                                            --exclude PROGRAM
                                            --exclude PID
                                            --exclude LEGACY_MSGHDR
                         --key time)\n"
         )

    url("http:/xxxxx.its.auckland.ac.nz:9200/_bulk<http://xxxxx.its.auckland.ac.nz:9200/_bulk>")
  );
};


clearly I a missing something as usual!

looking at the plugin code I assume that the field names are all prefixed with 'checkpoint.’ if I wanted to shorten this to ‘cp.’ then should I invoke it as checkpoint-parser(‘cp.’ ) ?



> On 18/05/2020, at 5:50 PM, Balazs Scheidler <bazsi77 at gmail.com<mailto:bazsi77 at gmail.com>> wrote:
>
> Hi,
>
> although the entire sample is not visible in your email, we have a checkpoint parser that fixes rfc5424 logs from checkpoint as it is using an incorrect format. More specifically the SDATA field is using colons instead of equal signs as in this example:
>
>  <134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update"; flags:"150784"; ifdir:"inbound"; logid:"160571424"; loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}"; origin:"192.168.32.91"; originsicname:"CN=GW91,O=Domain2_Server..cuggd3"; sequencenum:"1"; time:"1521645925"; version:"5"; auth_method:"Machine Authentication (Active Directory)"; auth_status:"Successful Login"; authentication_trial:"this is a reauthentication for session 9a026bba"; client_name:"Active Directory Query"; client_version:"R80.10"; domain_name:"spec.mgmt"; endpoint_ip:"192.168.32.69"; identity_src:"AD Query"; identity_type:"machine"; product:"Identity Awareness"; snid:"9a026bba"; src:"192.168.32.69"; src_machine_group:"All Machines"; src_machine_name:"yonatanad";]
>
> we have a checkpoint-parser() component that parses and fixes this into name-value pairs since 3.21, but with fixes in 3.26:
>
> https://github.com/syslog-ng/syslog-ng/blob/master/scl/checkpoint/plugin.conf
>
> To use that parser you will need to receive logs with flags(no-parse) and then apply the checkpoint-parser() as defined above.
>
> Also, there's a component in syslog-ng that receives messages on all sane channels and parses/fixes them automatically, this is the "default-network-drivers()" source, which apart from receiving on tcp/udp/ssl + rfc5424 will properly parse cisco/checkpoint/etc logs by applying application specific filtering/parsing rules, called application adapters. For instance the adapter for checkpoint looks like this:
>
> application checkpoint[syslog-raw] {
>     filter {
>         # "syslog" format
>         message("^(<[0-9]{1,3}>)1 .* CheckPoint ") or
>         # "splunk" format
>         message('^time=[0-9]+\|hostname=[a-zA-Z0-9-]+\|product=Firewall');
>     };
>     parser { checkpoint-parser(); };
> };
>
> This tells syslog-ng that it should look for checkpoint messages in the "syslog-raw" topic (e.g. stream of messages), apply checkpoint-parser() to log messages where the filter above matches. As you can see it will identify both "syslog" and "splunk" style messages emitted from checkpoint.
>
> On Mon, May 18, 2020 at 4:15 AM Russell Fulton <r.fulton at auckland.ac.nz<mailto:r.fulton at auckland.ac.nz>> wrote:
> I am trying to ingest logs from a checkpoint firewall over a tcp connection:
>
> source s_checkpoint {
>     network( transport("tcp") flags(no-multi-line) port(2514) keep-alive(yes) flags(syslog-protocol) );
> };
>
> # added flags(syslog-protocol) on suggest of checkpoint docs for syslog-ng
>
> destination d_debug {
>             file( "/data/russell//test.log"   );
>          };
>
>
> log {
>     source( s_checkpoint );
>     destination( d_debug );
> };
>
>
> This is what turns up in the output:
>
> May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -…..
>
> Any ideas on what is going on or how to find out?
>
> Russell
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
>
> --
> Bazsi
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.balabit.com/wiki/syslog-ng-faq

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200519/98198008/attachment-0001.html>


More information about the syslog-ng mailing list