[syslog-ng] [FORGED] getting "Error processing log message" for checkpoint logs

Tue May 19 06:07:49 UTC 2020

Yes.

On Tue, May 19, 2020, 06:25 Russell Fulton <r.fulton at auckland.ac.nz> wrote:

> Thanks Balazs.
>
> I am using 3.26 and have amended my config to disable the default parser
> and specify the new one:
>
> source s_checkpoint {
>     network( transport("tcp") flags(no-multi-line) port(2514)
> flags(no-parse) );
> };
>
> log {
>     source( s_checkpoint );
>     parser { checkpoint-parser(); };
>     destination( d_debug );
> };
>
> I no long get the parser errors and the output in the debug file looks
> like this:
>
> May 19 04:07:16 smartlog01 CheckPoint[14153]: [action:"Accept";
> conn_direction:"Outgoing"; flags:"6422528"; ifdir:"inbound";
> ifname:"bond2.440"; logid:"321"; loguid:"{0x0,0x0,0x0,0x0}";
> origin:"172.19.2.38"; originsicname:"CN=br-cpf4,O=
> smartcenter01.net.auckland.ac.nz.qgmmde"; sequencenum:"47";
> time:"1589861236"; version:"5"; __policy_id_tag:"product=VPN-1 &
> FireWall-1[db_tag={22CAD420-4CBB-2340-917B-1C32C546DEB1};mgmt=smartcenter01;date=1589757719;policy_name=Border_Policy\]";
> connection_luuid:"01001000-8300-00c0-5ec3-5b3800000001";
> hll_key:"533041188277717813"; product:"Application Control"; method:"GET";
> resource:"
> http://storage.googleapis.com/update-delta/gkmgaooipdjhmangpemjhigmamcehddo/81.235.200/80.230.200/3d83bc1481c09741d8bd52f2cef8af3e2206f2f0b5c816b46d9725be61977a6f.crxd";
> ]
>
> so they are clearly going through the parser.
>
> when I log it to elastic search I just get the standard fields with the
> body in MESSAGE, no parsed N V pairs.
> destination d_elastic_checkpoint {
>
>   elasticsearch_http(
>     index("checkpoint_${YEAR}.${MONTH}.${DAY}")
>    type("_doc")
>    persist-name("checkpoint")
>    template("$(format-json --scope nv-pairs --exclude HOST_FROM
>
>
>                                             --exclude HOST
>
>
>                                             --exclude SOURCE
>
>
>                                             --exclude PROGRAM
>
>
>                                             --exclude PID
>
>
>                                             --exclude LEGACY_MSGHDR
>
>
>                          --key time)\n"
>          )
>
>     url("http:/xxxxx.its.auckland.ac.nz:9200/_bulk")
>   );
> };
>
>
> clearly I a missing something as usual!
>
> looking at the plugin code I assume that the field names are all prefixed
> with 'checkpoint.’ if I wanted to shorten this to ‘cp.’ then should I
> invoke it as checkpoint-parser(‘cp.’ ) ?
>
>
>
> > On 18/05/2020, at 5:50 PM, Balazs Scheidler <bazsi77 at gmail.com> wrote:
> >
> > Hi,
> >
> > although the entire sample is not visible in your email, we have a
> checkpoint parser that fixes rfc5424 logs from checkpoint as it is using an
> incorrect format. More specifically the SDATA field is using colons instead
> of equal signs as in this example:
> >
> >  <134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update";
> flags:"150784"; ifdir:"inbound"; logid:"160571424";
> loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}"; origin:"192.168.32.91";
> originsicname:"CN=GW91,O=Domain2_Server..cuggd3"; sequencenum:"1";
> time:"1521645925"; version:"5"; auth_method:"Machine Authentication (Active
> Directory)"; auth_status:"Successful Login"; authentication_trial:"this is
> a reauthentication for session 9a026bba"; client_name:"Active Directory
> Query"; client_version:"R80.10"; domain_name:"spec.mgmt";
> endpoint_ip:"192.168.32.69"; identity_src:"AD Query";
> identity_type:"machine"; product:"Identity Awareness"; snid:"9a026bba";
> src:"192.168.32.69"; src_machine_group:"All Machines";
> src_machine_name:"yonatanad";]
> >
> > we have a checkpoint-parser() component that parses and fixes this into
> name-value pairs since 3.21, but with fixes in 3.26:
> >
> >
> https://github.com/syslog-ng/syslog-ng/blob/master/scl/checkpoint/plugin.conf
> >
> > To use that parser you will need to receive logs with flags(no-parse)
> and then apply the checkpoint-parser() as defined above.
> >
> > Also, there's a component in syslog-ng that receives messages on all
> sane channels and parses/fixes them automatically, this is the
> "default-network-drivers()" source, which apart from receiving on
> tcp/udp/ssl + rfc5424 will properly parse cisco/checkpoint/etc logs by
> applying application specific filtering/parsing rules, called application
> adapters. For instance the adapter for checkpoint looks like this:
> >
> > application checkpoint[syslog-raw] {
> >     filter {
> >         # "syslog" format
> >         message("^(<[0-9]{1,3}>)1 .* CheckPoint ") or
> >         # "splunk" format
> >
>  message('^time=[0-9]+\|hostname=[a-zA-Z0-9-]+\|product=Firewall');
> >     };
> >     parser { checkpoint-parser(); };
> > };
> >
> > This tells syslog-ng that it should look for checkpoint messages in the
> "syslog-raw" topic (e.g. stream of messages), apply checkpoint-parser() to
> log messages where the filter above matches. As you can see it will
> identify both "syslog" and "splunk" style messages emitted from checkpoint.
> >
> > On Mon, May 18, 2020 at 4:15 AM Russell Fulton <r.fulton at auckland.ac.nz>
> wrote:
> > I am trying to ingest logs from a checkpoint firewall over a tcp
> connection:
> >
> > source s_checkpoint {
> >     network( transport("tcp") flags(no-multi-line) port(2514)
> keep-alive(yes) flags(syslog-protocol) );
> > };
> >
> > # added flags(syslog-protocol) on suggest of checkpoint docs for
> syslog-ng
> >
> > destination d_debug {
> >             file( "/data/russell//test.log"   );
> >          };
> >
> >
> > log {
> >     source( s_checkpoint );
> >     destination( d_debug );
> > };
> >
> >
> > This is what turns up in the output:
> >
> > May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log
> message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -…..
> >
> > Any ideas on what is going on or how to find out?
> >
> > Russell
> >
> >
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
> >
> >
> > --
> > Bazsi
> >
> ______________________________________________________________________________
> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> > Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
> >
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200519/874b7731/attachment.html>