Greetings - I have been trying to create a very basic filter that looks up a string in the incoming log. However, it would not match any filter and would go to the default filter. I have tried both match() and message(), neither worked for me. Please let me know if you think I am missing something. Following are the chunks from the syslog-ng.conf source s_net { # All syslog traffic on port 514 - this is direct from network devices. udp(port (514)); network(transport("tcp") max-connections(20000) log_iw_size(100000000) ); # tags("fortigate", "cisco", "default") ); }; filter f_discreg { message("default send string") }; log { source(s_net); filter(f_dlptracker); destination(d_dlptracker); flags(final); }; Here is the log from the test I ran. [2020-12-02T22:00:36+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_tanium' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Outgoing message; destination='d_fallback#0', message='2020-12-02T22:00:46+00:00 172.17.236.3 default send string\x0a' Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m@cummins.com<mailto:saqib.m@cummins.com> Cummins Inc.
Hello! I just took a quick look on the config and on the internal logs. Couldn't be that "f_discreg" is not included in a log path? Another thing that could be is parsing: your "s_net" source will try to parse incoming messages as either BSD or Syslog format. If you've just tested the filter by sending in the message "default send string", then syslog-ng will parse it, which results that MSG macro will not contain "default send string". With trace level logging you can debug how does the message being parsed (-t command line option), and to disable parsing on the source side use the flags("no-parse") option. See an example: [2020-12-03T09:14:51.016256] Incoming log entry; line='default send string' [2020-12-03T09:14:51.016290] Initial message parsing follows; [2020-12-03T09:14:51.016322] Setting value; name='PROGRAM', value='default', msg='0x7f24a8005f30' [2020-12-03T09:14:51.016334] Setting value; name='LEGACY_MSGHDR', value='default ', msg='0x7f24a8005f30' [2020-12-03T09:14:51.016341] Setting value; name='MESSAGE', value='send string', msg='0x7f24a8005f30' Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Saqib M <saqib.m@cummins.com> Sent: Thursday, December 3, 2020 1:48 To: syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu> Subject: [syslog-ng] Filter matching not working CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Greetings – I have been trying to create a very basic filter that looks up a string in the incoming log. However, it would not match any filter and would go to the default filter. I have tried both match() and message(), neither worked for me. Please let me know if you think I am missing something. Following are the chunks from the syslog-ng.conf source s_net { # All syslog traffic on port 514 - this is direct from network devices. udp(port (514)); network(transport("tcp") max-connections(20000) log_iw_size(100000000) ); # tags("fortigate", "cisco", "default") ); }; filter f_discreg { message("default send string") }; log { source(s_net); filter(f_dlptracker); destination(d_dlptracker); flags(final); }; Here is the log from the test I ran. [2020-12-02T22:00:36+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_tanium' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Outgoing message; destination='d_fallback#0', message='2020-12-02T22:00:46+00:00 172.17.236.3 default send string\x0a' Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m@cummins.com<mailto:saqib.m@cummins.com> Cummins Inc.
Thank you for getting back on this. I am using the following command but I don't see the parsing logs. /opt/syslog-ng/sbin/syslog-ng -e -F -d -v -t This is what I saw. [2020-12-03T15:03:06+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-03T15:03:06+0000] Filter rule evaluation begins; filter_rule='f_discreg' [2020-12-03T15:03:06+0000] Filter node evaluation result; filter_result='not-match' [2020-12-03T15:03:06+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_discreg' [2020-12-03T15:03:06+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-03T15:03:06+0000] Filter node evaluation result; filter_result='not-match' [2020-12-03T15:03:06+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-03T15:03:06+0000] Outgoing message; destination='d_fallback#0', message='2020-12-03T15:03:06+00:00 172.17.236.3 default send string\x0a' Also, the f_discreg is in a log path. log { source(s_net); filter(f_discreg); destination(d_discard); flags(final); }; Could there be other ways to look into it a bit further to see what's going wrong? Thank you Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m@cummins.com<mailto:saqib.m@cummins.com> Cummins Inc. From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Gabor Nagy (gnagy) Sent: Thursday, December 3, 2020 3:17 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Filter matching not working External Sender Hello! I just took a quick look on the config and on the internal logs. Couldn't be that "f_discreg" is not included in a log path? Another thing that could be is parsing: your "s_net" source will try to parse incoming messages as either BSD or Syslog format. If you've just tested the filter by sending in the message "default send string", then syslog-ng will parse it, which results that MSG macro will not contain "default send string". With trace level logging you can debug how does the message being parsed (-t command line option), and to disable parsing on the source side use the flags("no-parse") option. See an example: [2020-12-03T09:14:51.016256] Incoming log entry; line='default send string' [2020-12-03T09:14:51.016290] Initial message parsing follows; [2020-12-03T09:14:51.016322] Setting value; name='PROGRAM', value='default', msg='0x7f24a8005f30' [2020-12-03T09:14:51.016334] Setting value; name='LEGACY_MSGHDR', value='default ', msg='0x7f24a8005f30' [2020-12-03T09:14:51.016341] Setting value; name='MESSAGE', value='send string', msg='0x7f24a8005f30' Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Saqib M <saqib.m@cummins.com<mailto:saqib.m@cummins.com>> Sent: Thursday, December 3, 2020 1:48 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Filter matching not working CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Greetings - I have been trying to create a very basic filter that looks up a string in the incoming log. However, it would not match any filter and would go to the default filter. I have tried both match() and message(), neither worked for me. Please let me know if you think I am missing something. Following are the chunks from the syslog-ng.conf source s_net { # All syslog traffic on port 514 - this is direct from network devices. udp(port (514)); network(transport("tcp") max-connections(20000) log_iw_size(100000000) ); # tags("fortigate", "cisco", "default") ); }; filter f_discreg { message("default send string") }; log { source(s_net); filter(f_dlptracker); destination(d_dlptracker); flags(final); }; Here is the log from the test I ran. [2020-12-02T22:00:36+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_tanium' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Outgoing message; destination='d_fallback#0', message='2020-12-02T22:00:46+00:00 172.17.236.3 default send string\x0a' Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m@cummins.com<mailto:saqib.m@cummins.com> Cummins Inc.
I still can't see more debug logs in your internal log. I guess you are using an older version of syslog-ng. Unfortunately trace level debugging can only be used if syslog-ng was compiled with trace level support. Can you tell us what version of syslog-ng are you using? Can you test your filter with the following BSD format message, please? <13>Dec 4 10:27:20 localhost myprogram: default send string Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Saqib M <saqib.m@cummins.com> Sent: Thursday, December 3, 2020 16:26 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Filter matching not working CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Thank you for getting back on this. I am using the following command but I don’t see the parsing logs. /opt/syslog-ng/sbin/syslog-ng -e -F -d -v -t This is what I saw. [2020-12-03T15:03:06+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-03T15:03:06+0000] Filter rule evaluation begins; filter_rule='f_discreg' [2020-12-03T15:03:06+0000] Filter node evaluation result; filter_result='not-match' [2020-12-03T15:03:06+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_discreg' [2020-12-03T15:03:06+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-03T15:03:06+0000] Filter node evaluation result; filter_result='not-match' [2020-12-03T15:03:06+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-03T15:03:06+0000] Outgoing message; destination='d_fallback#0', message='2020-12-03T15:03:06+00:00 172.17.236.3 default send string\x0a' Also, the f_discreg is in a log path. log { source(s_net); filter(f_discreg); destination(d_discard); flags(final); }; Could there be other ways to look into it a bit further to see what’s going wrong? Thank you Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m@cummins.com<mailto:saqib.m@cummins.com> Cummins Inc. From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Gabor Nagy (gnagy) Sent: Thursday, December 3, 2020 3:17 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Filter matching not working External Sender Hello! I just took a quick look on the config and on the internal logs. Couldn't be that "f_discreg" is not included in a log path? Another thing that could be is parsing: your "s_net" source will try to parse incoming messages as either BSD or Syslog format. If you've just tested the filter by sending in the message "default send string", then syslog-ng will parse it, which results that MSG macro will not contain "default send string". With trace level logging you can debug how does the message being parsed (-t command line option), and to disable parsing on the source side use the flags("no-parse") option. See an example: [2020-12-03T09:14:51.016256] Incoming log entry; line='default send string' [2020-12-03T09:14:51.016290] Initial message parsing follows; [2020-12-03T09:14:51.016322] Setting value; name='PROGRAM', value='default', msg='0x7f24a8005f30' [2020-12-03T09:14:51.016334] Setting value; name='LEGACY_MSGHDR', value='default ', msg='0x7f24a8005f30' [2020-12-03T09:14:51.016341] Setting value; name='MESSAGE', value='send string', msg='0x7f24a8005f30' Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Saqib M <saqib.m@cummins.com<mailto:saqib.m@cummins.com>> Sent: Thursday, December 3, 2020 1:48 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Filter matching not working CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Greetings – I have been trying to create a very basic filter that looks up a string in the incoming log. However, it would not match any filter and would go to the default filter. I have tried both match() and message(), neither worked for me. Please let me know if you think I am missing something. Following are the chunks from the syslog-ng.conf source s_net { # All syslog traffic on port 514 - this is direct from network devices. udp(port (514)); network(transport("tcp") max-connections(20000) log_iw_size(100000000) ); # tags("fortigate", "cisco", "default") ); }; filter f_discreg { message("default send string") }; log { source(s_net); filter(f_dlptracker); destination(d_dlptracker); flags(final); }; Here is the log from the test I ran. [2020-12-02T22:00:36+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_tanium' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Outgoing message; destination='d_fallback#0', message='2020-12-02T22:00:46+00:00 172.17.236.3 default send string\x0a' Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m@cummins.com<mailto:saqib.m@cummins.com> Cummins Inc.
Hello Saqib M, I think your problem still can be just a small misconfiguration. It would be really good if you can share the exact version of Syslog-ng, your whole configuration and multiple example messages. So we won't accidentally overlook something. (If your configuration contains private information you can either scramble it or send it in private to one of us.) Example: (What was Gábor referring to) With a fresh Syslog-ng, you should see these additional debug messages in the output: Incoming log entry; line='default send string' Initial message parsing follows; Setting value; name='PROGRAM', value='default', msg='0x7f8214006fc0' Setting value; name='LEGACY_MSGHDR', value='default ', msg='0x7f8214006fc0' Setting value; name='MESSAGE', value='send string', msg='0x7f8214006fc0'
From this you can see, that the "udp" and "network" sources (without any other instruction) are trying to parse the incoming message as an RFC5424 message, which will result to unexpected values in certain macros. For example, in this particular case the "MESSAGE" macro only contains the string "send string", that is why "default send string" do not match on it.
Best regards, Laci ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> Sent: Friday, December 4, 2020 10:29 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Filter matching not working CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. I still can't see more debug logs in your internal log. I guess you are using an older version of syslog-ng. Unfortunately trace level debugging can only be used if syslog-ng was compiled with trace level support. Can you tell us what version of syslog-ng are you using? Can you test your filter with the following BSD format message, please? <13>Dec 4 10:27:20 localhost myprogram: default send string Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Saqib M <saqib.m@cummins.com> Sent: Thursday, December 3, 2020 16:26 To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Filter matching not working CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Thank you for getting back on this. I am using the following command but I don’t see the parsing logs. /opt/syslog-ng/sbin/syslog-ng -e -F -d -v -t This is what I saw. [2020-12-03T15:03:06+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-03T15:03:06+0000] Filter rule evaluation begins; filter_rule='f_discreg' [2020-12-03T15:03:06+0000] Filter node evaluation result; filter_result='not-match' [2020-12-03T15:03:06+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_discreg' [2020-12-03T15:03:06+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-03T15:03:06+0000] Filter node evaluation result; filter_result='not-match' [2020-12-03T15:03:06+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-03T15:03:06+0000] Outgoing message; destination='d_fallback#0', message='2020-12-03T15:03:06+00:00 172.17.236.3 default send string\x0a' Also, the f_discreg is in a log path. log { source(s_net); filter(f_discreg); destination(d_discard); flags(final); }; Could there be other ways to look into it a bit further to see what’s going wrong? Thank you Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m@cummins.com<mailto:saqib.m@cummins.com> Cummins Inc. From: syslog-ng <syslog-ng-bounces@lists.balabit.hu> On Behalf Of Gabor Nagy (gnagy) Sent: Thursday, December 3, 2020 3:17 AM To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu> Subject: Re: [syslog-ng] Filter matching not working External Sender Hello! I just took a quick look on the config and on the internal logs. Couldn't be that "f_discreg" is not included in a log path? Another thing that could be is parsing: your "s_net" source will try to parse incoming messages as either BSD or Syslog format. If you've just tested the filter by sending in the message "default send string", then syslog-ng will parse it, which results that MSG macro will not contain "default send string". With trace level logging you can debug how does the message being parsed (-t command line option), and to disable parsing on the source side use the flags("no-parse") option. See an example: [2020-12-03T09:14:51.016256] Incoming log entry; line='default send string' [2020-12-03T09:14:51.016290] Initial message parsing follows; [2020-12-03T09:14:51.016322] Setting value; name='PROGRAM', value='default', msg='0x7f24a8005f30' [2020-12-03T09:14:51.016334] Setting value; name='LEGACY_MSGHDR', value='default ', msg='0x7f24a8005f30' [2020-12-03T09:14:51.016341] Setting value; name='MESSAGE', value='send string', msg='0x7f24a8005f30' Regards, Gabor ________________________________ From: syslog-ng <syslog-ng-bounces@lists.balabit.hu<mailto:syslog-ng-bounces@lists.balabit.hu>> on behalf of Saqib M <saqib.m@cummins.com<mailto:saqib.m@cummins.com>> Sent: Thursday, December 3, 2020 1:48 To: syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu> <syslog-ng@lists.balabit.hu<mailto:syslog-ng@lists.balabit.hu>> Subject: [syslog-ng] Filter matching not working CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe. Greetings – I have been trying to create a very basic filter that looks up a string in the incoming log. However, it would not match any filter and would go to the default filter. I have tried both match() and message(), neither worked for me. Please let me know if you think I am missing something. Following are the chunks from the syslog-ng.conf source s_net { # All syslog traffic on port 514 - this is direct from network devices. udp(port (514)); network(transport("tcp") max-connections(20000) log_iw_size(100000000) ); # tags("fortigate", "cisco", "default") ); }; filter f_discreg { message("default send string") }; log { source(s_net); filter(f_dlptracker); destination(d_dlptracker); flags(final); }; Here is the log from the test I ran. [2020-12-02T22:00:36+0000] Incoming log entry; source='s_net#0', line='default send string' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_tanium' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_palo_alto' [2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match' [2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker' [2020-12-02T22:00:46+0000] Outgoing message; destination='d_fallback#0', message='2020-12-02T22:00:46+00:00 172.17.236.3 default send string\x0a' Regards, Saqib M Cybersecurity Co-op Global Cybersecurity Technologies Email: saqib.m@cummins.com<mailto:saqib.m@cummins.com> Cummins Inc.
Dear Saqib, Perhaps it would be easier if you could share a debug bundle with us. Syslog-ng OSE has a debug bundle generator script included, which should take care of the process. If your OS's package doesn't include it, you can download it from: https://raw.githubusercontent.com/syslog-ng/syslog-ng/master/contrib/syslog-... If you you add executable rights to it, and run it with the options: # ./syslog-ng-debun -r -d -t 600 -p It will run syslog-ng in debug mode for 10 minutes, and will create a packet capture of the logs coming in on ports 514 (udp+tcp) and 601 (tcp) plus the DNS traffic. (You should also make sure that your problematic device actually sends traffic to the syslog server in those 10 minutes!) If you'd like, you can look at the bundle's contents to verify you are not sending us anything sensitive. If you send the bundle to us (Laszlo, Gabor and/or me) privately, that will also help in preserving your deployment's information. Thank you! Best Regards, János Szigetvári -- Janos SZIGETVARI RHCE, License no. 150-053-692 <https://www.redhat.com/rhtapps/verify/?certId=150-053-692> LinkedIn: linkedin.com/in/janosszigetvari __@__˚V˚ Make the switch to open (source) applications, protocols, formats now: - windows -> Linux, iexplore -> Firefox, msoffice -> LibreOffice - msn -> jabber protocol (Pidgin, Google Talk) - mp3 -> ogg, wmv -> ogg, jpg -> png, doc/xls/ppt -> odt/ods/odp Laszlo Szemere (lszemere) <Laszlo.Szemere@oneidentity.com> ezt írta (időpont: 2020. dec. 7., H, 8:22):
Hello Saqib M,
I think your problem still can be just a small misconfiguration. It would be really good if you can share the exact version of Syslog-ng, your whole configuration and multiple example messages. So we won't accidentally overlook something. (If your configuration contains private information you can either scramble it or send it in private to one of us.)
Example: (What was Gábor referring to) With a fresh Syslog-ng, you should see these additional debug messages in the output:
Incoming log entry; line='default send string' Initial message parsing follows; Setting value; name='PROGRAM', value='default', msg='0x7f8214006fc0' Setting value; name='LEGACY_MSGHDR', value='default ', msg= '0x7f8214006fc0' Setting value; name='MESSAGE', value='send string', msg='0x7f8214006fc0'
From this you can see, that the "udp" and "network" sources (without any other instruction) are trying to parse the incoming message as an RFC5424 message, which will result to unexpected values in certain macros. For example, in this particular case the "MESSAGE" macro only contains the string "send string", that is why "default send string" do not match on it.
Best regards, Laci
------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Gabor Nagy (gnagy) <Gabor.Nagy@oneidentity.com> *Sent:* Friday, December 4, 2020 10:29 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Filter matching not working
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
I still can't see more debug logs in your internal log. I guess you are using an older version of syslog-ng. Unfortunately trace level debugging can only be used if syslog-ng was compiled with trace level support. Can you tell us what version of syslog-ng are you using?
Can you test your filter with the following BSD format message, please? <13>Dec 4 10:27:20 localhost myprogram: default send string
Regards, Gabor ------------------------------ *From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Saqib M <saqib.m@cummins.com> *Sent:* Thursday, December 3, 2020 16:26 *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Filter matching not working
CAUTION: This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Thank you for getting back on this. I am using the following command but I don’t see the parsing logs.
/opt/syslog-ng/sbin/syslog-ng -e -F -d -v -t
This is what I saw.
[2020-12-03T15:03:06+0000] Incoming log entry; source='s_net#0', line='default send string'
[2020-12-03T15:03:06+0000] Filter rule evaluation begins; filter_rule='f_discreg'
[2020-12-03T15:03:06+0000] Filter node evaluation result; filter_result='not-match'
[2020-12-03T15:03:06+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_discreg'
[2020-12-03T15:03:06+0000] Filter rule evaluation begins; filter_rule='f_dlptracker'
[2020-12-03T15:03:06+0000] Filter node evaluation result; filter_result='not-match'
[2020-12-03T15:03:06+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker'
[2020-12-03T15:03:06+0000] Outgoing message; destination='d_fallback#0', message='2020-12-03T15:03:06+00:00 172.17.236.3 default send string\x0a'
Also, the f_discreg is in a log path.
log { source(s_net); filter(f_discreg); destination(d_discard); flags(final); };
Could there be other ways to look into it a bit further to see what’s going wrong?
Thank you
Regards,
Saqib M
*Cybersecurity Co-op*
Global Cybersecurity Technologies
Email: saqib.m@cummins.com
Cummins Inc.
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> *On Behalf Of *Gabor Nagy (gnagy) *Sent:* Thursday, December 3, 2020 3:17 AM *To:* Syslog-ng users' and developers' mailing list < syslog-ng@lists.balabit.hu> *Subject:* Re: [syslog-ng] Filter matching not working
*External Sender*
Hello!
I just took a quick look on the config and on the internal logs. Couldn't be that "f_discreg" is not included in a log path?
Another thing that could be is parsing: your "s_net" source will try to parse incoming messages as either BSD or Syslog format. If you've just tested the filter by sending in the message "default send string", then syslog-ng will parse it, which results that MSG macro will not contain "default send string".
With trace level logging you can debug how does the message being parsed (-t command line option), and to disable parsing on the source side use the flags("no-parse") option.
See an example: [2020-12-03T09:14:51.016256] Incoming log entry; line='default send string'
[2020-12-03T09:14:51.016290] Initial message parsing follows;
[2020-12-03T09:14:51.016322] Setting value; name='PROGRAM', value='default', msg='0x7f24a8005f30'
[2020-12-03T09:14:51.016334] Setting value; name='LEGACY_MSGHDR', value='default ', msg='0x7f24a8005f30'
[2020-12-03T09:14:51.016341] Setting value; name='MESSAGE', value='send string', msg='0x7f24a8005f30'
Regards, Gabor ------------------------------
*From:* syslog-ng <syslog-ng-bounces@lists.balabit.hu> on behalf of Saqib M <saqib.m@cummins.com> *Sent:* Thursday, December 3, 2020 1:48 *To:* syslog-ng@lists.balabit.hu <syslog-ng@lists.balabit.hu> *Subject:* [syslog-ng] Filter matching not working
*CAUTION:* This email originated from outside of the organization. Do not follow guidance, click links, or open attachments unless you recognize the sender and know the content is safe.
Greetings –
I have been trying to create a very basic filter that looks up a string in the incoming log. However, it would not match any filter and would go to the default filter. I have tried both match() and message(), neither worked for me. Please let me know if you think I am missing something.
*Following are the chunks from the syslog-ng.conf*
source s_net {
# All syslog traffic on port 514 - this is direct from network devices.
udp(port (514));
network(transport("tcp") max-connections(20000) log_iw_size(100000000) ); # tags("fortigate", "cisco", "default") );
};
filter f_discreg { message("default send string") };
log { source(s_net); filter(f_dlptracker); destination(d_dlptracker); flags(final); };
*Here is the log from the test I ran.*
[2020-12-02T22:00:36+0000] Incoming log entry; source='s_net#0', line='default send string'
[2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR'
[2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_tanium'
[2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_palo_alto'
[2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match'
[2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match'
[2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match'
[2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR'
[2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match', filter_type='OR'
[2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_palo_alto'
[2020-12-02T22:00:46+0000] Filter rule evaluation begins; filter_rule='f_dlptracker'
[2020-12-02T22:00:46+0000] Filter node evaluation result; filter_result='not-match'
[2020-12-02T22:00:46+0000] Filter rule evaluation result; filter_result='not-match', filter_rule='f_dlptracker'
[2020-12-02T22:00:46+0000] Outgoing message; destination='d_fallback#0', message='2020-12-02T22:00:46+00:00 172.17.236.3 default send string\x0a'
Regards,
Saqib M
*Cybersecurity Co-op*
Global Cybersecurity Technologies
Email: saqib.m@cummins.com
Cummins Inc.
______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.balabit.com/wiki/syslog-ng-faq
participants (4)
-
Gabor Nagy (gnagy)
-
Laszlo Szemere (lszemere)
-
Saqib M
-
SZIGETVÁRI János