This is what my current configuration looks like now (disabled TLS to allow for debugging): http://www.murpe.com/syslog-ng-v3.conf2.txt -- M ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alan McKinnon [Alan.McKinnon@is.co.za] Sent: Tuesday, September 21, 2010 10:35 AM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] [Bug 93] New: filter() functionality between 2.1 to 3.0 not consistent I think you are running into the changed definitions of match() between 2.0 and 3.0 http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guid... admin-en.html/reference_filters.html especially match() and message() Of your samples, the 3rd and 4th should work. What is the full config that applies to how you use this filter? Are you perhaps applying two filters? Be careful with that - filters are always ANDed in a log statement, not ORed. On Tuesday 21 September 2010 16:16:57 bugzilla@bugzilla.balabit.com wrote:

https://bugzilla.balabit.com/show_bug.cgi?id=93

Summary: filter() functionality between 2.1 to 3.0 not consistent Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: mworsham@scires.com Type of the Report: bug Estimated Hours: 0.0

In the older syslong-ng v2.1, this line works perfectly:

filter M_audit { not match("Audit daemon rotating log files"); };

Under 3.0.8, none of the following are working (if added one line at a time) and the daemon restarted:

filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY) flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files")); };

What I am looking to do is if any incoming data has the following keywords (i.e. "Audit daemon rotating log files") being detected, it should be filtered (i.e. dropped), and then not show up in the actual message log file.

For example, if I go over on the syslog-ng client and do 'logger daemon': - The older v2.1 syslog-ng server detects the embedded keyword and drops the message. <-- This is correct - On the v3.0.8, the message passes through and found in the data file. <-- This is incorrect

Attempted to use the following URLs to correct the formatting, but it still not working: - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gu ide-admin-en.html/configuring_filters.html - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gu ide-admin-en.html/reference_macros.html

-- Alan McKinnon Systems Engineer^W Technician Infrastructure Services Internet Solutions +27 11 575 7585 Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers@is.co.za and a copy will be emailed to you. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

One of my co-workers lent a pair of eyes found something rather unique between the two versions I hadn't looked at. Right now, the v3.0.8 build, we are using has the following filter configuration modified from the v2.1 build: filter M_audit { not message("Audit daemon rotating log files"); }; filter M_repeat { not message("last message repeated"); }; filter M_stats { not message("Log statistics"); }; Now, if I were to login to the syslog-ng client server and do the following 'logger last message repeated', the entire message is dropped from actually showing up on the remote syslog-ng server (which it should). However, if I do 'logger repeated' (a portion cut from the actual full message that is to be filtered), then the remote syslog-ng server will record it. The filters, in the way they are configured, are NOT allowing for portions of the messages to be detected -- it's either the entire message is matched or none of it, not pieces. What do I need to do to allow the pieces to be detected and filtered out just like the full message? -- M ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Tuesday, September 21, 2010 10:48 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] [Bug 93] New: filter() functionality between 2.1 to 3.0 not consistent This is what my current configuration looks like now (disabled TLS to allow for debugging): http://www.murpe.com/syslog-ng-v3.conf2.txt -- M ________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alan McKinnon [Alan.McKinnon@is.co.za] Sent: Tuesday, September 21, 2010 10:35 AM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] [Bug 93] New: filter() functionality between 2.1 to 3.0 not consistent I think you are running into the changed definitions of match() between 2.0 and 3.0 http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-guid... admin-en.html/reference_filters.html especially match() and message() Of your samples, the 3rd and 4th should work. What is the full config that applies to how you use this filter? Are you perhaps applying two filters? Be careful with that - filters are always ANDed in a log statement, not ORed. On Tuesday 21 September 2010 16:16:57 bugzilla@bugzilla.balabit.com wrote:

https://bugzilla.balabit.com/show_bug.cgi?id=93

Summary: filter() functionality between 2.1 to 3.0 not consistent Product: syslog-ng Version: 3.0.x Platform: PC OS/Version: Linux Status: NEW Severity: major Priority: unspecified Component: syslog-ng AssignedTo: bazsi@balabit.hu ReportedBy: mworsham@scires.com Type of the Report: bug Estimated Hours: 0.0

In the older syslong-ng v2.1, this line works perfectly:

filter M_audit { not match("Audit daemon rotating log files"); };

Under 3.0.8, none of the following are working (if added one line at a time) and the daemon restarted:

filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY) flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files")); };

What I am looking to do is if any incoming data has the following keywords (i.e. "Audit daemon rotating log files") being detected, it should be filtered (i.e. dropped), and then not show up in the actual message log file.

For example, if I go over on the syslog-ng client and do 'logger daemon': - The older v2.1 syslog-ng server detects the embedded keyword and drops the message. <-- This is correct - On the v3.0.8, the message passes through and found in the data file. <-- This is incorrect

Attempted to use the following URLs to correct the formatting, but it still not working: - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gu ide-admin-en.html/configuring_filters.html - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gu ide-admin-en.html/reference_macros.html

-- Alan McKinnon Systems Engineer^W Technician Infrastructure Services Internet Solutions +27 11 575 7585 Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers@is.co.za and a copy will be emailed to you. ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements. ________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments. EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

You seem to have simple logic errors, I do not think regexes are your problem. However it will be good to make sure that you configure pcre support when you built syslog-ng Here's the relevant parts of your config. filter M_audit { not message("Audit daemon rotating log files" flags(ignore- case)); }; filter M_repeat { not message("last message repeated" flags(ignore-case)); }; filter M_stats { not message("Log statistics" flags(ignore-case)); }; log { source(s_general); filter(M_audit); filter(M_repeat); filter(M_stats); destination(d_general); }; First, filters are ANDed. So it's a case of a log is sent to d_general only if M_audit is true AND M_repeat is true AND M_stats is true. It is not OR! Secondly, M_repeat is false if the string "last message repeated" IS in the log. It does not say that M_repeat is true if the string is not in the log. Those things are very different - some simply boolean algebra will show it. Moving onto your log statement, all 3 filters must be true for the message to be logged to the destination. This evaluates to none of the three strings must be present. When you run "logger last message repeated", the message() evaluates to true, the not makes it false and the log does not match. These booleans get tricky as you do not have the entire spectrum of boolean operations available in the syslog-ng syntax. In combinations, it gets even worse and quickly becomes unmaintainable. And negative matches are even worse than that. I find a much better approach is to make individual tests on messages, discard the ones you are not interested in and then write positive match rules to log what is left (all of those are then by definition logs you want to keep). Try this approach instead: filter M_audit { message("Audit daemon rotating log files" flags(ignore- case)); }; filter M_repeat { message("last message repeated" flags(ignore-case)); }; filter M_stats { message("Log statistics" flags(ignore-case)); }; log { source(s_general); filter(M_audit); flags(final); }; log { source(s_general); filter(M_repeat); flags(final); }; log { source(s_general); filter(M_stats); flags(final); }; Add here filter and log statement for messages you wish to keep. On Tuesday 21 September 2010 22:38:59 Worsham, Michael wrote:

One of my co-workers lent a pair of eyes found something rather unique between the two versions I hadn't looked at. Right now, the v3.0.8 build, we are using has the following filter configuration modified from the v2.1 build:

filter M_audit { not message("Audit daemon rotating log files"); }; filter M_repeat { not message("last message repeated"); }; filter M_stats { not message("Log statistics"); }; Now, if I were to login to the syslog-ng client server and do the following 'logger last message repeated', the entire message is dropped from actually showing up on the remote syslog-ng server (which it should). However, if I do 'logger repeated' (a portion cut from the actual full message that is to be filtered), then the remote syslog-ng server will record it.

The filters, in the way they are configured, are NOT allowing for portions of the messages to be detected -- it's either the entire message is matched or none of it, not pieces. What do I need to do to allow the pieces to be detected and filtered out just like the full message?

-- M

________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Worsham, Michael Sent: Tuesday, September 21, 2010 10:48 AM To: Syslog-ng users' and developers' mailing list Subject: Re: [syslog-ng] [Bug 93] New: filter() functionality between 2.1 to 3.0 not consistent

This is what my current configuration looks like now (disabled TLS to allow for debugging):

http://www.murpe.com/syslog-ng-v3.conf2.txt

-- M

________________________________ From: syslog-ng-bounces@lists.balabit.hu [syslog-ng-bounces@lists.balabit.hu] On Behalf Of Alan McKinnon [Alan.McKinnon@is.co.za] Sent: Tuesday, September 21, 2010 10:35 AM To: syslog-ng@lists.balabit.hu Subject: Re: [syslog-ng] [Bug 93] New: filter() functionality between 2.1 to 3.0 not consistent

I think you are running into the changed definitions of match() between 2.0 and 3.0

http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui de- admin-en.html/reference_filters.html

especially match() and message()

Of your samples, the 3rd and 4th should work. What is the full config that applies to how you use this filter? Are you perhaps applying two filters? Be careful with that - filters are always ANDed in a log statement, not ORed.

On Tuesday 21 September 2010 16:16:57 bugzilla@bugzilla.balabit.com wrote:

...

https://bugzilla.balabit.com/show_bug.cgi?id=93

Summary: filter() functionality between 2.1 to 3.0 not

consistent Product: syslog-ng

Version: 3.0.x

Platform: PC

OS/Version: Linux

Status: NEW

Severity: major Priority: unspecified

Component: syslog-ng

AssignedTo: bazsi@balabit.hu ReportedBy: mworsham@scires.com

Type of the Report: bug

Estimated Hours: 0.0

In the older syslong-ng v2.1, this line works perfectly:

filter M_audit { not match("Audit daemon rotating log files"); };

Under 3.0.8, none of the following are working (if added one line at a time) and the daemon restarted:

filter M_audit { not match("Audit daemon rotating log files" value("MSGONLY") flags(ignore-case)); }; filter M_audit { not match("MSGONLY" value("Audit daemon rotating log files") flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY) flags(ignore-case)); }; filter M_audit { not match("Audit daemon rotating log files" value(MSGONLY)); }; filter M_audit

{ not match("MSGONLY" value("Audit daemon rotating log files")); };

What I am looking to do is if any incoming data has the following keywords (i.e. "Audit daemon rotating log files") being detected, it should be filtered (i.e. dropped), and then not show up in the actual message log file.

For example, if I go over on the syslog-ng client and do 'logger daemon': - The older v2.1 syslog-ng server detects the embedded keyword and

drops the message. <-- This is correct - On the v3.0.8, the message passes through and found in the data file. <-- This is incorrect

Attempted to use the following URLs to correct the formatting, but it still not working: - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-g u ide-admin-en.html/configuring_filters.html - http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-g u ide-admin-en.html/reference_macros.html

-- Alan McKinnon Systems Engineer^W Technician Infrastructure Services Internet Solutions

+27 11 575 7585

Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm . Should you not have Web access, send a mail to disclaimers@is.co.za and a copy will be emailed to you. __________________________________________________________________________ ____ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html

________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.

EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

________________________________ CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.

EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.

-- Alan McKinnon Systems Engineer^W Technician Infrastructure Services Internet Solutions +27 11 575 7585 Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers@is.co.za and a copy will be emailed to you.