getting "Error processing log message" for checkpoint logs
I am trying to ingest logs from a checkpoint firewall over a tcp connection: source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) keep-alive(yes) flags(syslog-protocol) ); }; # added flags(syslog-protocol) on suggest of checkpoint docs for syslog-ng destination d_debug { file( "/data/russell//test.log" ); }; log { source( s_checkpoint ); destination( d_debug ); }; This is what turns up in the output: May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -….. Any ideas on what is going on or how to find out? Russell
Hi, although the entire sample is not visible in your email, we have a checkpoint parser that fixes rfc5424 logs from checkpoint as it is using an incorrect format. More specifically the SDATA field is using colons instead of equal signs as in this example: <134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update"; flags:"150784"; ifdir:"inbound"; logid:"160571424"; loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}"; origin:"192.168.32.91"; originsicname:"CN=GW91,O=Domain2_Server..cuggd3"; sequencenum:"1"; time:"1521645925"; version:"5"; auth_method:"Machine Authentication (Active Directory)"; auth_status:"Successful Login"; authentication_trial:"this is a reauthentication for session 9a026bba"; client_name:"Active Directory Query"; client_version:"R80.10"; domain_name:"spec.mgmt"; endpoint_ip:"192.168.32.69"; identity_src:"AD Query"; identity_type:"machine"; product:"Identity Awareness"; snid:"9a026bba"; src:"192.168.32.69"; src_machine_group:"All Machines"; src_machine_name:"yonatanad";] we have a checkpoint-parser() component that parses and fixes this into name-value pairs since 3.21, but with fixes in 3.26: https://github.com/syslog-ng/syslog-ng/blob/master/scl/checkpoint/plugin.con... To use that parser you will need to receive logs with flags(no-parse) and then apply the checkpoint-parser() as defined above. Also, there's a component in syslog-ng that receives messages on all sane channels and parses/fixes them automatically, this is the "default-network-drivers()" source, which apart from receiving on tcp/udp/ssl + rfc5424 will properly parse cisco/checkpoint/etc logs by applying application specific filtering/parsing rules, called application adapters. For instance the adapter for checkpoint looks like this: application checkpoint[syslog-raw] { filter { # "syslog" format message("^(<[0-9]{1,3}>)1 .* CheckPoint ") or # "splunk" format message('^time=[0-9]+\|hostname=[a-zA-Z0-9-]+\|product=Firewall'); }; parser { checkpoint-parser(); }; }; This tells syslog-ng that it should look for checkpoint messages in the "syslog-raw" topic (e.g. stream of messages), apply checkpoint-parser() to log messages where the filter above matches. As you can see it will identify both "syslog" and "splunk" style messages emitted from checkpoint. On Mon, May 18, 2020 at 4:15 AM Russell Fulton <r.fulton@auckland.ac.nz> wrote:
I am trying to ingest logs from a checkpoint firewall over a tcp connection:
source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) keep-alive(yes) flags(syslog-protocol) ); };
# added flags(syslog-protocol) on suggest of checkpoint docs for syslog-ng
destination d_debug { file( "/data/russell//test.log" ); };
log { source( s_checkpoint ); destination( d_debug ); };
This is what turns up in the output:
May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -…..
Any ideas on what is going on or how to find out?
Russell
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
Thanks Balazs. I am using 3.26 and have amended my config to disable the default parser and specify the new one: source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) flags(no-parse) ); }; log { source( s_checkpoint ); parser { checkpoint-parser(); }; destination( d_debug ); }; I no long get the parser errors and the output in the debug file looks like this: May 19 04:07:16 smartlog01 CheckPoint[14153]: [action:"Accept"; conn_direction:"Outgoing"; flags:"6422528"; ifdir:"inbound"; ifname:"bond2.440"; logid:"321"; loguid:"{0x0,0x0,0x0,0x0}"; origin:"172.19.2.38"; originsicname:"CN=br-cpf4,O=smartcenter01.net.auckland.ac.nz.qgmmde"; sequencenum:"47"; time:"1589861236"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={22CAD420-4CBB-2340-917B-1C32C546DEB1};mgmt=smartcenter01;date=1589757719;policy_name=Border_Policy\]"; connection_luuid:"01001000-8300-00c0-5ec3-5b3800000001"; hll_key:"533041188277717813"; product:"Application Control"; method:"GET"; resource:"http://storage.googleapis.com/update-delta/gkmgaooipdjhmangpemjhigmamcehddo/..."; ] so they are clearly going through the parser. when I log it to elastic search I just get the standard fields with the body in MESSAGE, no parsed N V pairs. destination d_elastic_checkpoint { elasticsearch_http( index("checkpoint_${YEAR}.${MONTH}.${DAY}") type("_doc") persist-name("checkpoint") template("$(format-json --scope nv-pairs --exclude HOST_FROM --exclude HOST --exclude SOURCE --exclude PROGRAM --exclude PID --exclude LEGACY_MSGHDR --key time)\n" ) url("http:/xxxxx.its.auckland.ac.nz:9200/_bulk") ); }; clearly I a missing something as usual! looking at the plugin code I assume that the field names are all prefixed with 'checkpoint.’ if I wanted to shorten this to ‘cp.’ then should I invoke it as checkpoint-parser(‘cp.’ ) ?
On 18/05/2020, at 5:50 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
Hi,
although the entire sample is not visible in your email, we have a checkpoint parser that fixes rfc5424 logs from checkpoint as it is using an incorrect format. More specifically the SDATA field is using colons instead of equal signs as in this example:
<134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update"; flags:"150784"; ifdir:"inbound"; logid:"160571424"; loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}"; origin:"192.168.32.91"; originsicname:"CN=GW91,O=Domain2_Server..cuggd3"; sequencenum:"1"; time:"1521645925"; version:"5"; auth_method:"Machine Authentication (Active Directory)"; auth_status:"Successful Login"; authentication_trial:"this is a reauthentication for session 9a026bba"; client_name:"Active Directory Query"; client_version:"R80.10"; domain_name:"spec.mgmt"; endpoint_ip:"192.168.32.69"; identity_src:"AD Query"; identity_type:"machine"; product:"Identity Awareness"; snid:"9a026bba"; src:"192.168.32.69"; src_machine_group:"All Machines"; src_machine_name:"yonatanad";]
we have a checkpoint-parser() component that parses and fixes this into name-value pairs since 3.21, but with fixes in 3.26:
https://github.com/syslog-ng/syslog-ng/blob/master/scl/checkpoint/plugin.con...
To use that parser you will need to receive logs with flags(no-parse) and then apply the checkpoint-parser() as defined above.
Also, there's a component in syslog-ng that receives messages on all sane channels and parses/fixes them automatically, this is the "default-network-drivers()" source, which apart from receiving on tcp/udp/ssl + rfc5424 will properly parse cisco/checkpoint/etc logs by applying application specific filtering/parsing rules, called application adapters. For instance the adapter for checkpoint looks like this:
application checkpoint[syslog-raw] { filter { # "syslog" format message("^(<[0-9]{1,3}>)1 .* CheckPoint ") or # "splunk" format message('^time=[0-9]+\|hostname=[a-zA-Z0-9-]+\|product=Firewall'); }; parser { checkpoint-parser(); }; };
This tells syslog-ng that it should look for checkpoint messages in the "syslog-raw" topic (e.g. stream of messages), apply checkpoint-parser() to log messages where the filter above matches. As you can see it will identify both "syslog" and "splunk" style messages emitted from checkpoint.
On Mon, May 18, 2020 at 4:15 AM Russell Fulton <r.fulton@auckland.ac.nz> wrote: I am trying to ingest logs from a checkpoint firewall over a tcp connection:
source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) keep-alive(yes) flags(syslog-protocol) ); };
# added flags(syslog-protocol) on suggest of checkpoint docs for syslog-ng
destination d_debug { file( "/data/russell//test.log" ); };
log { source( s_checkpoint ); destination( d_debug ); };
This is what turns up in the output:
May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -…..
Any ideas on what is going on or how to find out?
Russell
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Yes. On Tue, May 19, 2020, 06:25 Russell Fulton <r.fulton@auckland.ac.nz> wrote:
Thanks Balazs.
I am using 3.26 and have amended my config to disable the default parser and specify the new one:
source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) flags(no-parse) ); };
log { source( s_checkpoint ); parser { checkpoint-parser(); }; destination( d_debug ); };
I no long get the parser errors and the output in the debug file looks like this:
May 19 04:07:16 smartlog01 CheckPoint[14153]: [action:"Accept"; conn_direction:"Outgoing"; flags:"6422528"; ifdir:"inbound"; ifname:"bond2.440"; logid:"321"; loguid:"{0x0,0x0,0x0,0x0}"; origin:"172.19.2.38"; originsicname:"CN=br-cpf4,O= smartcenter01.net.auckland.ac.nz.qgmmde"; sequencenum:"47"; time:"1589861236"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={22CAD420-4CBB-2340-917B-1C32C546DEB1};mgmt=smartcenter01;date=1589757719;policy_name=Border_Policy\]"; connection_luuid:"01001000-8300-00c0-5ec3-5b3800000001"; hll_key:"533041188277717813"; product:"Application Control"; method:"GET"; resource:" http://storage.googleapis.com/update-delta/gkmgaooipdjhmangpemjhigmamcehddo/..."; ]
so they are clearly going through the parser.
when I log it to elastic search I just get the standard fields with the body in MESSAGE, no parsed N V pairs. destination d_elastic_checkpoint {
elasticsearch_http( index("checkpoint_${YEAR}.${MONTH}.${DAY}") type("_doc") persist-name("checkpoint") template("$(format-json --scope nv-pairs --exclude HOST_FROM
--exclude HOST
--exclude SOURCE
--exclude PROGRAM
--exclude PID
--exclude LEGACY_MSGHDR
--key time)\n" )
url("http:/xxxxx.its.auckland.ac.nz:9200/_bulk") ); };
clearly I a missing something as usual!
looking at the plugin code I assume that the field names are all prefixed with 'checkpoint.’ if I wanted to shorten this to ‘cp.’ then should I invoke it as checkpoint-parser(‘cp.’ ) ?
On 18/05/2020, at 5:50 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
Hi,
although the entire sample is not visible in your email, we have a checkpoint parser that fixes rfc5424 logs from checkpoint as it is using an incorrect format. More specifically the SDATA field is using colons instead of equal signs as in this example:
<134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update"; flags:"150784"; ifdir:"inbound"; logid:"160571424"; loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}"; origin:"192.168.32.91"; originsicname:"CN=GW91,O=Domain2_Server..cuggd3"; sequencenum:"1"; time:"1521645925"; version:"5"; auth_method:"Machine Authentication (Active Directory)"; auth_status:"Successful Login"; authentication_trial:"this is a reauthentication for session 9a026bba"; client_name:"Active Directory Query"; client_version:"R80.10"; domain_name:"spec.mgmt"; endpoint_ip:"192.168.32.69"; identity_src:"AD Query"; identity_type:"machine"; product:"Identity Awareness"; snid:"9a026bba"; src:"192.168.32.69"; src_machine_group:"All Machines"; src_machine_name:"yonatanad";]
we have a checkpoint-parser() component that parses and fixes this into name-value pairs since 3.21, but with fixes in 3.26:
https://github.com/syslog-ng/syslog-ng/blob/master/scl/checkpoint/plugin.con...
To use that parser you will need to receive logs with flags(no-parse)
and then apply the checkpoint-parser() as defined above.
Also, there's a component in syslog-ng that receives messages on all
sane channels and parses/fixes them automatically, this is the "default-network-drivers()" source, which apart from receiving on tcp/udp/ssl + rfc5424 will properly parse cisco/checkpoint/etc logs by applying application specific filtering/parsing rules, called application adapters. For instance the adapter for checkpoint looks like this:
application checkpoint[syslog-raw] { filter { # "syslog" format message("^(<[0-9]{1,3}>)1 .* CheckPoint ") or # "splunk" format
message('^time=[0-9]+\|hostname=[a-zA-Z0-9-]+\|product=Firewall');
}; parser { checkpoint-parser(); }; };
This tells syslog-ng that it should look for checkpoint messages in the
"syslog-raw" topic (e.g. stream of messages), apply checkpoint-parser() to log messages where the filter above matches. As you can see it will identify both "syslog" and "splunk" style messages emitted from checkpoint.
On Mon, May 18, 2020 at 4:15 AM Russell Fulton <r.fulton@auckland.ac.nz>
wrote:
I am trying to ingest logs from a checkpoint firewall over a tcp connection:
source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) keep-alive(yes) flags(syslog-protocol) ); };
# added flags(syslog-protocol) on suggest of checkpoint docs for syslog-ng
destination d_debug { file( "/data/russell//test.log" ); };
log { source( s_checkpoint ); destination( d_debug ); };
This is what turns up in the output:
May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -…..
Any ideas on what is going on or how to find out?
Russell
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
Russell@fulton.nz On 19/05/2020, at 6:08 PM, Balazs Scheidler <bazsi77@gmail.com> wrote: Yes. My bad! There were two questions! The one you missed was that when i pass the logs to elastic search i don’t get the parsed n v pairs. I get the standard syslog record with all the details in MESSAGE THe question about the prefix was an afterthought, one i should have squashed! Russell On Tue, May 19, 2020, 06:25 Russell Fulton <r.fulton@auckland.ac.nz<mailto:r.fulton@auckland.ac.nz>> wrote: Thanks Balazs. I am using 3.26 and have amended my config to disable the default parser and specify the new one: source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) flags(no-parse) ); }; log { source( s_checkpoint ); parser { checkpoint-parser(); }; destination( d_debug ); }; I no long get the parser errors and the output in the debug file looks like this: May 19 04:07:16 smartlog01 CheckPoint[14153]: [action:"Accept"; conn_direction:"Outgoing"; flags:"6422528"; ifdir:"inbound"; ifname:"bond2.440"; logid:"321"; loguid:"{0x0,0x0,0x0,0x0}"; origin:"172.19.2.38"; originsicname:"CN=br-cpf4,O=smartcenter01.net.auckland.ac<http://smartcenter01.net.auckland.ac>.nz.qgmmde"; sequencenum:"47"; time:"1589861236"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={22CAD420-4CBB-2340-917B-1C32C546DEB1};mgmt=smartcenter01;date=1589757719;policy_name=Border_Policy\]"; connection_luuid:"01001000-8300-00c0-5ec3-5b3800000001"; hll_key:"533041188277717813"; product:"Application Control"; method:"GET"; resource:"http://storage.googleapis.com/update-delta/gkmgaooipdjhmangpemjhigmamcehddo/..."; ] so they are clearly going through the parser. when I log it to elastic search I just get the standard fields with the body in MESSAGE, no parsed N V pairs. destination d_elastic_checkpoint { elasticsearch_http( index("checkpoint_${YEAR}.${MONTH}.${DAY}") type("_doc") persist-name("checkpoint") template("$(format-json --scope nv-pairs --exclude HOST_FROM --exclude HOST --exclude SOURCE --exclude PROGRAM --exclude PID --exclude LEGACY_MSGHDR --key time)\n" ) url("http:/xxxxx.its.auckland.ac.nz:9200/_bulk<http://xxxxx.its.auckland.ac.nz:9200/_bulk>") ); }; clearly I a missing something as usual! looking at the plugin code I assume that the field names are all prefixed with 'checkpoint.’ if I wanted to shorten this to ‘cp.’ then should I invoke it as checkpoint-parser(‘cp.’ ) ?
On 18/05/2020, at 5:50 PM, Balazs Scheidler <bazsi77@gmail.com<mailto:bazsi77@gmail.com>> wrote:
Hi,
although the entire sample is not visible in your email, we have a checkpoint parser that fixes rfc5424 logs from checkpoint as it is using an incorrect format. More specifically the SDATA field is using colons instead of equal signs as in this example:
<134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update"; flags:"150784"; ifdir:"inbound"; logid:"160571424"; loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}"; origin:"192.168.32.91"; originsicname:"CN=GW91,O=Domain2_Server..cuggd3"; sequencenum:"1"; time:"1521645925"; version:"5"; auth_method:"Machine Authentication (Active Directory)"; auth_status:"Successful Login"; authentication_trial:"this is a reauthentication for session 9a026bba"; client_name:"Active Directory Query"; client_version:"R80.10"; domain_name:"spec.mgmt"; endpoint_ip:"192.168.32.69"; identity_src:"AD Query"; identity_type:"machine"; product:"Identity Awareness"; snid:"9a026bba"; src:"192.168.32.69"; src_machine_group:"All Machines"; src_machine_name:"yonatanad";]
we have a checkpoint-parser() component that parses and fixes this into name-value pairs since 3.21, but with fixes in 3.26:
https://github.com/syslog-ng/syslog-ng/blob/master/scl/checkpoint/plugin.con...
To use that parser you will need to receive logs with flags(no-parse) and then apply the checkpoint-parser() as defined above.
Also, there's a component in syslog-ng that receives messages on all sane channels and parses/fixes them automatically, this is the "default-network-drivers()" source, which apart from receiving on tcp/udp/ssl + rfc5424 will properly parse cisco/checkpoint/etc logs by applying application specific filtering/parsing rules, called application adapters. For instance the adapter for checkpoint looks like this:
application checkpoint[syslog-raw] { filter { # "syslog" format message("^(<[0-9]{1,3}>)1 .* CheckPoint ") or # "splunk" format message('^time=[0-9]+\|hostname=[a-zA-Z0-9-]+\|product=Firewall'); }; parser { checkpoint-parser(); }; };
This tells syslog-ng that it should look for checkpoint messages in the "syslog-raw" topic (e.g. stream of messages), apply checkpoint-parser() to log messages where the filter above matches. As you can see it will identify both "syslog" and "splunk" style messages emitted from checkpoint.
On Mon, May 18, 2020 at 4:15 AM Russell Fulton <r.fulton@auckland.ac.nz<mailto:r.fulton@auckland.ac.nz>> wrote: I am trying to ingest logs from a checkpoint firewall over a tcp connection:
source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) keep-alive(yes) flags(syslog-protocol) ); };
# added flags(syslog-protocol) on suggest of checkpoint docs for syslog-ng
destination d_debug { file( "/data/russell//test.log" ); };
log { source( s_checkpoint ); destination( d_debug ); };
This is what turns up in the output:
May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -…..
Any ideas on what is going on or how to find out?
Russell
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
scope should be all-nv-pairs or use a wildcard that includes your prefix. On Tue, May 19, 2020, 21:10 Russell Fulton <r.fulton@auckland.ac.nz> wrote:
Russell@fulton.nz
On 19/05/2020, at 6:08 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
Yes.
My bad! There were two questions!
The one you missed was that when i pass the logs to elastic search i don’t get the parsed n v pairs. I get the standard syslog record with all the details in MESSAGE
THe question about the prefix was an afterthought, one i should have squashed!
Russell
On Tue, May 19, 2020, 06:25 Russell Fulton <r.fulton@auckland.ac.nz> wrote:
Thanks Balazs.
I am using 3.26 and have amended my config to disable the default parser and specify the new one:
source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) flags(no-parse) ); };
log { source( s_checkpoint ); parser { checkpoint-parser(); }; destination( d_debug ); };
I no long get the parser errors and the output in the debug file looks like this:
May 19 04:07:16 smartlog01 CheckPoint[14153]: [action:"Accept"; conn_direction:"Outgoing"; flags:"6422528"; ifdir:"inbound"; ifname:"bond2.440"; logid:"321"; loguid:"{0x0,0x0,0x0,0x0}"; origin:"172.19.2.38"; originsicname:"CN=br-cpf4,O= smartcenter01.net.auckland.ac.nz.qgmmde"; sequencenum:"47"; time:"1589861236"; version:"5"; __policy_id_tag:"product=VPN-1 & FireWall-1[db_tag={22CAD420-4CBB-2340-917B-1C32C546DEB1};mgmt=smartcenter01;date=1589757719;policy_name=Border_Policy\]"; connection_luuid:"01001000-8300-00c0-5ec3-5b3800000001"; hll_key:"533041188277717813"; product:"Application Control"; method:"GET"; resource:" http://storage.googleapis.com/update-delta/gkmgaooipdjhmangpemjhigmamcehddo/..."; ]
so they are clearly going through the parser.
when I log it to elastic search I just get the standard fields with the body in MESSAGE, no parsed N V pairs. destination d_elastic_checkpoint {
elasticsearch_http( index("checkpoint_${YEAR}.${MONTH}.${DAY}") type("_doc") persist-name("checkpoint") template("$(format-json --scope nv-pairs --exclude HOST_FROM
--exclude HOST
--exclude SOURCE
--exclude PROGRAM
--exclude PID
--exclude LEGACY_MSGHDR
--key time)\n" )
url("http:/xxxxx.its.auckland.ac.nz:9200/_bulk") ); };
clearly I a missing something as usual!
looking at the plugin code I assume that the field names are all prefixed with 'checkpoint.’ if I wanted to shorten this to ‘cp.’ then should I invoke it as checkpoint-parser(‘cp.’ ) ?
On 18/05/2020, at 5:50 PM, Balazs Scheidler <bazsi77@gmail.com> wrote:
Hi,
although the entire sample is not visible in your email, we have a checkpoint parser that fixes rfc5424 logs from checkpoint as it is using an incorrect format. More specifically the SDATA field is using colons instead of equal signs as in this example:
<134>1 2018-03-21 17:25:25 MDS-72 CheckPoint 13752 - [action:"Update"; flags:"150784"; ifdir:"inbound"; logid:"160571424"; loguid:"{0x5ab27965,0x0,0x5b20a8c0,0x7d5707b6}"; origin:"192.168.32.91"; originsicname:"CN=GW91,O=Domain2_Server..cuggd3"; sequencenum:"1"; time:"1521645925"; version:"5"; auth_method:"Machine Authentication (Active Directory)"; auth_status:"Successful Login"; authentication_trial:"this is a reauthentication for session 9a026bba"; client_name:"Active Directory Query"; client_version:"R80.10"; domain_name:"spec.mgmt"; endpoint_ip:"192.168.32.69"; identity_src:"AD Query"; identity_type:"machine"; product:"Identity Awareness"; snid:"9a026bba"; src:"192.168.32.69"; src_machine_group:"All Machines"; src_machine_name:"yonatanad";]
we have a checkpoint-parser() component that parses and fixes this into name-value pairs since 3.21, but with fixes in 3.26:
https://github.com/syslog-ng/syslog-ng/blob/master/scl/checkpoint/plugin.con...
To use that parser you will need to receive logs with flags(no-parse)
and then apply the checkpoint-parser() as defined above.
Also, there's a component in syslog-ng that receives messages on all
sane channels and parses/fixes them automatically, this is the "default-network-drivers()" source, which apart from receiving on tcp/udp/ssl + rfc5424 will properly parse cisco/checkpoint/etc logs by applying application specific filtering/parsing rules, called application adapters. For instance the adapter for checkpoint looks like this:
application checkpoint[syslog-raw] { filter { # "syslog" format message("^(<[0-9]{1,3}>)1 .* CheckPoint ") or # "splunk" format
message('^time=[0-9]+\|hostname=[a-zA-Z0-9-]+\|product=Firewall');
}; parser { checkpoint-parser(); }; };
This tells syslog-ng that it should look for checkpoint messages in the
"syslog-raw" topic (e.g. stream of messages), apply checkpoint-parser() to log messages where the filter above matches. As you can see it will identify both "syslog" and "splunk" style messages emitted from checkpoint.
On Mon, May 18, 2020 at 4:15 AM Russell Fulton <r.fulton@auckland.ac.nz>
wrote:
I am trying to ingest logs from a checkpoint firewall over a tcp connection:
source s_checkpoint { network( transport("tcp") flags(no-multi-line) port(2514) keep-alive(yes) flags(syslog-protocol) ); };
# added flags(syslog-protocol) on suggest of checkpoint docs for syslog-ng
destination d_debug { file( "/data/russell//test.log" ); };
log { source( s_checkpoint ); destination( d_debug ); };
This is what turns up in the output:
May 18 12:10:42 secmgrprd02 syslog-ng[1555]: Error processing log message: <134>1 2020-05-17T23:33:27Z smartlog01 CheckPoint 25651 -…..
Any ideas on what is going on or how to find out?
Russell
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
-- Bazsi
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (2)
-
Balazs Scheidler
-
Russell Fulton