[syslog-ng] [FORGED] Re: [FORGED] getting "Error processing log message" for checkpoint logs
Balazs Scheidler
bazsi77 at gmail.com
Wed May 20 05:12:57 UTC 2020
scope should be all-nv-pairs or use a wildcard that includes your prefix.
On Tue, May 19, 2020, 21:10 Russell Fulton <r.fulton at auckland.ac.nz> wrote:
>
>
> Russell at fulton.nz
>
> On 19/05/2020, at 6:08 PM, Balazs Scheidler <bazsi77 at gmail.com> wrote:
>
>
> Yes.
>
> My bad! There were two questions!
>
>
> The one you missed was that when i pass the logs to elastic search i don’t
> get the parsed n v pairs. I get the standard syslog record with all the
> details in MESSAGE
>
> THe question about the prefix was an afterthought, one i should have
> squashed!
>
> Russell
>
>
> On Tue, May 19, 2020, 06:25 Russell Fulton <r.fulton at auckland.ac.nz>
> wrote:
>
>> Thanks Balazs.
>>
>> I am using 3.26 and have amended my config to disable the default parser
>> and specify the new one:
>>
>> source s_checkpoint {
>> network( transport("tcp") flags(no-multi-line) port(2514)
>> flags(no-parse) );
>> };
>>
>> log {
>> source( s_checkpoint );
>> parser { checkpoint-parser(); };
>> destination( d_debug );
>> };
>>
>> I no long get the parser errors and the output in the debug file looks
>> like this:
>>
>> May 19 04:07:16 smartlog01 CheckPoint[14153]: [action:"Accept";
>> conn_direction:"Outgoing"; flags:"6422528"; ifdir:"inbound";
>> ifname:"bond2.440"; logid:"321"; loguid:"{0x0,0x0,0x0,0x0}";
>> origin:"172.19.2.38"; originsicname:"CN=br-cpf4,O=
>> smartcenter01.net.auckland.ac.nz.qgmmde"; sequencenum:"47";
>> time:"1589861236"; version:"5"; __policy_id_tag:"product=VPN-1 &
>> FireWall-1[db_tag={22CAD420-4CBB-2340-917B-1C32C546DEB1};mgmt=smartcenter01;date=1589757719;policy_name=Border_Policy\]";
>> connection_luuid:"01001000-8300-00c0-5ec3-5b3800000001";
>> hll_key:"533041188277717813"; product:"Application Control"; method:"GET";
>> resource:"
>> http://storage.googleapis.com/update-delta/gkmgaooipdjhmangpemjhigmamcehddo/81.235.200/80.230.200/3d83bc1481c09741d8bd52f2cef8af3e2206f2f0b5c816b46d9725be61977a6f.crxd";
>> ]
>>
>> so they are clearly going through the parser.
>>
>> when I log it to elastic search I just get the standard fields with the
>> body in MESSAGE, no parsed N V pairs.
>> destination d_elastic_checkpoint {
>>
>> elasticsearch_http(
>> index("checkpoint_${YEAR}.${MONTH}.${DAY}")
>> type("_doc")
>> persist-name("checkpoint")
>> template("$(format-json --scope nv-pairs --exclude HOST_FROM
>>
>>
>> --exclude HOST
>>
>>
>> --exclude SOURCE
>>
>>
>> --exclude PROGRAM
>>
>>
>> --exclude PID
>>
>>
>> --exclude LEGACY_MSGHDR
>>
>>
>> --key time)\n"
>> )
>>
>> url("http:/xxxxx.its.auckland.ac.nz:9200/_bulk")
>> );
>> };
>>
>>
>> clearly I a missing something as usual!
>>
>> looking at the plugin code I assume that the field names are all prefixed
>> with 'checkpoint.’ if I wanted to shorten this to ‘cp.’ then should I
>> invoke it as checkpoint-parser(‘cp.’ ) ?
>>
>>
>>
>> > On 18/05/2020, at 5:50 PM, Balazs Scheidler <bazsi77 at gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > although the entire sample is not visible in your email, we have a
>> checkpoint parser that fixes rfc5424 logs from checkpoint as it is using an
>> incorrect format. More specifically the SDATA field is using colons instead
>> of equal signs as in this example:
>> >
>> > <134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update";
>> flags:"150784"; ifdir:"inbound"; logid:"160571424";
>> loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}"; origin:"192.168.32.91";
>> originsicname:"CN=GW91,O=Domain2_Server..cuggd3"; sequencenum:"1";
>> time:"1521645925"; version:"5"; auth_method:"Machine Authentication (Active
>> Directory)"; auth_status:"Successful Login"; authentication_trial:"this is
>> a reauthentication for session 9a026bba"; client_name:"Active Directory
>> Query"; client_version:"R80.10"; domain_name:"spec.mgmt";
>> endpoint_ip:"192.168.32.69"; identity_src:"AD Query";
>> identity_type:"machine"; product:"Identity Awareness"; snid:"9a026bba";
>> src:"192.168.32.69"; src_machine_group:"All Machines";
>> src_machine_name:"yonatanad";]
>> >
>> > we have a checkpoint-parser() component that parses and fixes this into
>> name-value pairs since 3.21, but with fixes in 3.26:
>> >
>> >
>> https://github.com/syslog-ng/syslog-ng/blob/master/scl/checkpoint/plugin.conf
>> >
>> > To use that parser you will need to receive logs with flags(no-parse)
>> and then apply the checkpoint-parser() as defined above.
>> >
>> > Also, there's a component in syslog-ng that receives messages on all
>> sane channels and parses/fixes them automatically, this is the
>> "default-network-drivers()" source, which apart from receiving on
>> tcp/udp/ssl + rfc5424 will properly parse cisco/checkpoint/etc logs by
>> applying application specific filtering/parsing rules, called application
>> adapters. For instance the adapter for checkpoint looks like this:
>> >
>> > application checkpoint[syslog-raw] {
>> > filter {
>> > # "syslog" format
>> > message("^(<[0-9]{1,3}>)1 .* CheckPoint ") or
>> > # "splunk" format
>> >
>> message('^time=[0-9]+\|hostname=[a-zA-Z0-9-]+\|product=Firewall');
>> > };
>> > parser { checkpoint-parser(); };
>> > };
>> >
>> > This tells syslog-ng that it should look for checkpoint messages in the
>> "syslog-raw" topic (e.g. stream of messages), apply checkpoint-parser() to
>> log messages where the filter above matches. As you can see it will
>> identify both "syslog" and "splunk" style messages emitted from checkpoint.
>> >
>> > On Mon, May 18, 2020 at 4:15 AM Russell Fulton <r.fulton at auckland.ac.nz>
>> wrote:
>> > I am trying to ingest logs from a checkpoint firewall over a tcp
>> connection:
>> >
>> > source s_checkpoint {
>> > network( transport("tcp") flags(no-multi-line) port(2514)
>> keep-alive(yes) flags(syslog-protocol) );
>> > };
>> >
>> > # added flags(syslog-protocol) on suggest of checkpoint docs for
>> syslog-ng
>> >
>> > destination d_debug {
>> > file( "/data/russell//test.log" );
>> > };
>> >
>> >
>> > log {
>> > source( s_checkpoint );
>> > destination( d_debug );
>> > };
>> >
>> >
>> > This is what turns up in the output:
>> >
>> > May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log
>> message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -…..
>> >
>> > Any ideas on what is going on or how to find out?
>> >
>> > Russell
>> >
>> >
>> >
>> ______________________________________________________________________________
>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >
>> >
>> >
>> > --
>> > Bazsi
>> >
>> ______________________________________________________________________________
>> > Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> > Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> > FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>> >
>>
>>
>> ______________________________________________________________________________
>> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
>> Documentation:
>> http://www.balabit.com/support/documentation/?product=syslog-ng
>> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>>
>>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.balabit.com/wiki/syslog-ng-faq
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.balabit.hu/pipermail/syslog-ng/attachments/20200520/e9ff6a21/attachment.html>
More information about the syslog-ng
mailing list