[syslog-ng] [Bug 93] New: filter() functionality between 2.1 to 3.0 not consistent
Worsham, Michael
mworsham at SCIRES.COM
Wed Sep 22 03:44:35 CEST 2010
Since we are part of a govt project, we aren't allowed to compile code on the RHEL platforms as this will break our STIG/DISA requirements so we can only use RPM builds. (Yea, I know it sucks.).
As per the syslog-ng -V, it seems that PCRE is disabled in the RPM build so that option won't be working.
[root at syslog-svr sbin]# ./syslog-ng -V
syslog-ng 3.0.8
Installer-Version: 3.0.8
Revision: ssh+git://bazsi@git.balabit//var/scm/git/syslog-ng/syslog-ng-ose--mainline--3.0#master#c88009fc97b5014e330ecc8ead747691a05b7e97
Compile-Date: Jul 9 2010 08:32:54
Enable-Threads: on
Enable-Debug: off
Enable-GProf: off
Enable-Memtrace: off
Enable-Sun-STREAMS: off
Enable-Sun-Door: off
Enable-IPv6: on
Enable-Spoof-Source: on
Enable-TCP-Wrapper: off
Enable-SSL: on
Enable-SQL: on
Enable-Linux-Caps: on
Enable-Pcre: off
________________________________
From: syslog-ng-bounces at lists.balabit.hu [syslog-ng-bounces at lists.balabit.hu] On Behalf Of Alan McKinnon [Alan.McKinnon at is.co.za]
Sent: Tuesday, September 21, 2010 5:22 PM
To: syslog-ng at lists.balabit.hu
Subject: Re: [syslog-ng] [Bug 93] New: filter() functionality between 2.1 to 3.0 not consistent
You seem to have simple logic errors, I do not think regexes are your problem.
However it will be good to make sure that you configure pcre support when you
built syslog-ng
Here's the relevant parts of your config.
filter M_audit { not message("Audit daemon rotating log files" flags(ignore-
case)); };
filter M_repeat { not message("last message repeated" flags(ignore-case)); };
filter M_stats { not message("Log statistics" flags(ignore-case)); };
log { source(s_general);
filter(M_audit);
filter(M_repeat);
filter(M_stats);
destination(d_general);
};
First, filters are ANDed. So it's a case of a log is sent to d_general only if
M_audit is true AND M_repeat is true AND M_stats is true. It is not OR!
Secondly, M_repeat is false if the string "last message repeated" IS in the
log. It does not say that M_repeat is true if the string is not in the log.
Those things are very different - some simply boolean algebra will show it.
Moving onto your log statement, all 3 filters must be true for the message to
be logged to the destination. This evaluates to none of the three strings must
be present. When you run "logger last message repeated", the message()
evaluates to true, the not makes it false and the log does not match.
These booleans get tricky as you do not have the entire spectrum of boolean
operations available in the syslog-ng syntax. In combinations, it gets even
worse and quickly becomes unmaintainable. And negative matches are even worse
than that. I find a much better approach is to make individual tests on
messages, discard the ones you are not interested in and then write positive
match rules to log what is left (all of those are then by definition logs you
want to keep). Try this approach instead:
filter M_audit { message("Audit daemon rotating log files" flags(ignore-
case)); };
filter M_repeat { message("last message repeated" flags(ignore-case)); };
filter M_stats { message("Log statistics" flags(ignore-case)); };
log { source(s_general); filter(M_audit); flags(final); };
log { source(s_general); filter(M_repeat); flags(final); };
log { source(s_general); filter(M_stats); flags(final); };
Add here filter and log statement for messages you wish to keep.
On Tuesday 21 September 2010 22:38:59 Worsham, Michael wrote:
> One of my co-workers lent a pair of eyes found something rather unique
> between the two versions I hadn't looked at. Right now, the v3.0.8 build,
> we are using has the following filter configuration modified from the v2.1
> build:
>
> filter M_audit { not message("Audit daemon rotating log files"); };
> filter M_repeat { not message("last message repeated"); };
> filter M_stats { not message("Log statistics"); };
> Now, if I were to login to the syslog-ng client server and do the following
> 'logger last message repeated', the entire message is dropped from
> actually showing up on the remote syslog-ng server (which it should).
> However, if I do 'logger repeated' (a portion cut from the actual full
> message that is to be filtered), then the remote syslog-ng server will
> record it.
>
> The filters, in the way they are configured, are NOT allowing for portions
> of the messages to be detected -- it's either the entire message is
> matched or none of it, not pieces. What do I need to do to allow the
> pieces to be detected and filtered out just like the full message?
>
> -- M
>
> ________________________________
> From: syslog-ng-bounces at lists.balabit.hu
> [syslog-ng-bounces at lists.balabit.hu] On Behalf Of Worsham, Michael Sent:
> Tuesday, September 21, 2010 10:48 AM
> To: Syslog-ng users' and developers' mailing list
> Subject: Re: [syslog-ng] [Bug 93] New: filter() functionality between 2.1
> to 3.0 not consistent
>
> This is what my current configuration looks like now (disabled TLS to allow
> for debugging):
>
> http://www.murpe.com/syslog-ng-v3.conf2.txt
>
> -- M
>
> ________________________________
> From: syslog-ng-bounces at lists.balabit.hu
> [syslog-ng-bounces at lists.balabit.hu] On Behalf Of Alan McKinnon
> [Alan.McKinnon at is.co.za] Sent: Tuesday, September 21, 2010 10:35 AM
> To: syslog-ng at lists.balabit.hu
> Subject: Re: [syslog-ng] [Bug 93] New: filter() functionality between 2.1
> to 3.0 not consistent
>
> I think you are running into the changed definitions of match() between 2.0
> and 3.0
>
> http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-gui
> de- admin-en.html/reference_filters.html
>
> especially match() and message()
>
> Of your samples, the 3rd and 4th should work. What is the full config that
> applies to how you use this filter? Are you perhaps applying two filters?
> Be careful with that - filters are always ANDed in a log statement, not
> ORed.
>
> On Tuesday 21 September 2010 16:16:57 bugzilla at bugzilla.balabit.com wrote:
> > https://bugzilla.balabit.com/show_bug.cgi?id=93
> >
> > Summary: filter() functionality between 2.1 to 3.0 not
> >
> > consistent Product: syslog-ng
> >
> > Version: 3.0.x
> >
> > Platform: PC
> >
> > OS/Version: Linux
> >
> > Status: NEW
> >
> > Severity: major
> > Priority: unspecified
> >
> > Component: syslog-ng
> >
> > AssignedTo: bazsi at balabit.hu
> > ReportedBy: mworsham at scires.com
> >
> > Type of the Report: bug
> >
> > Estimated Hours: 0.0
> >
> > In the older syslong-ng v2.1, this line works perfectly:
> >
> > filter M_audit { not match("Audit daemon rotating log files"); };
> >
> > Under 3.0.8, none of the following are working (if added one line at a
> > time) and the daemon restarted:
> >
> > filter M_audit { not match("Audit daemon rotating log files"
> > value("MSGONLY") flags(ignore-case)); }; filter M_audit { not
> > match("MSGONLY" value("Audit daemon rotating log files")
> > flags(ignore-case)); }; filter M_audit { not match("Audit daemon
> > rotating log files" value(MSGONLY) flags(ignore-case)); }; filter
> > M_audit { not match("Audit daemon rotating log files" value(MSGONLY));
> > }; filter M_audit
> >
> > { not match("MSGONLY" value("Audit daemon rotating log files")); };
> >
> > What I am looking to do is if any incoming data has the following
> > keywords (i.e. "Audit daemon rotating log files") being detected, it
> > should be filtered (i.e. dropped), and then not show up in the actual
> > message log file.
> >
> > For example, if I go over on the syslog-ng client and do 'logger daemon':
> > - The older v2.1 syslog-ng server detects the embedded keyword and
> >
> > drops the message. <-- This is correct - On the v3.0.8, the message
> > passes through and found in the data file. <-- This is incorrect
> >
> > Attempted to use the following URLs to correct the formatting, but it
> > still not working: -
> > http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-g
> > u ide-admin-en.html/configuring_filters.html -
> > http://www.balabit.com/sites/default/files/documents/syslog-ng-ose-v3.1-g
> > u ide-admin-en.html/reference_macros.html
>
> --
> Alan McKinnon
> Systems Engineer^W Technician
> Infrastructure Services
> Internet Solutions
>
> +27 11 575 7585
>
> Please note: This email and its content are subject to the disclaimer as
> displayed at the following link
> http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm
> . Should you not have Web access, send a mail to disclaimers at is.co.za and a
> copy will be emailed to you.
> __________________________________________________________________________
> ____ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation:
> http://www.balabit.com/support/documentation/?product=syslog-ng FAQ:
> http://www.campin.net/syslog-ng/faq.html
>
>
> ________________________________
> CONFIDENTIALITY NOTICE: This email and any attachments are intended solely
> for the use of the named recipient(s). This email may contain confidential
> and/or proprietary information of Scientific Research Corporation. If you
> are not a named recipient, you are prohibited from reviewing, copying,
> using, disclosing or distributing to others the information in this email
> and attachments. If you believe you have received this email in error,
> please notify the sender immediately and permanently delete the email, any
> attachments, and all copies thereof from any drives or storage media and
> destroy any printouts of the email or attachments.
>
> EXPORT COMPLIANCE NOTICE: This email and any attachments may contain
> technical data subject to U.S export restrictions under the International
> Traffic in Arms Regulations (ITAR) or the Export Administration
> Regulations (EAR). Export or transfer of this technical data and/or
> related information to any foreign person(s) or entity(ies), either within
> the U.S. or outside of the U.S., may require advance export authorization
> by the appropriate U.S. Government agency prior to export or transfer. In
> addition, technical data may not be exported or transferred to certain
> countries or specified designated nationals identified by U.S. embargo
> controls without prior export authorization. By accepting this email and
> any attachments, all recipients confirm that they understand and will
> comply with all applicable ITAR, EAR and embargo compliance requirements.
>
> ________________________________
> CONFIDENTIALITY NOTICE: This email and any attachments are intended solely
> for the use of the named recipient(s). This email may contain confidential
> and/or proprietary information of Scientific Research Corporation. If you
> are not a named recipient, you are prohibited from reviewing, copying,
> using, disclosing or distributing to others the information in this email
> and attachments. If you believe you have received this email in error,
> please notify the sender immediately and permanently delete the email, any
> attachments, and all copies thereof from any drives or storage media and
> destroy any printouts of the email or attachments.
>
> EXPORT COMPLIANCE NOTICE: This email and any attachments may contain
> technical data subject to U.S export restrictions under the International
> Traffic in Arms Regulations (ITAR) or the Export Administration
> Regulations (EAR). Export or transfer of this technical data and/or
> related information to any foreign person(s) or entity(ies), either within
> the U.S. or outside of the U.S., may require advance export authorization
> by the appropriate U.S. Government agency prior to export or transfer. In
> addition, technical data may not be exported or transferred to certain
> countries or specified designated nationals identified by U.S. embargo
> controls without prior export authorization. By accepting this email and
> any attachments, all recipients confirm that they understand and will
> comply with all applicable ITAR, EAR and embargo compliance requirements.
--
Alan McKinnon
Systems Engineer^W Technician
Infrastructure Services
Internet Solutions
+27 11 575 7585
Please note: This email and its content are subject to the disclaimer as displayed at the following link http://www.is.co.za/legal/E-mail+Confidentiality+Notice+and+Disclaimer.htm. Should you not have Web access, send a mail to disclaimers at is.co.za and a copy will be emailed to you.
______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html
________________________________
CONFIDENTIALITY NOTICE: This email and any attachments are intended solely for the use of the named recipient(s). This email may contain confidential and/or proprietary information of Scientific Research Corporation. If you are not a named recipient, you are prohibited from reviewing, copying, using, disclosing or distributing to others the information in this email and attachments. If you believe you have received this email in error, please notify the sender immediately and permanently delete the email, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the email or attachments.
EXPORT COMPLIANCE NOTICE: This email and any attachments may contain technical data subject to U.S export restrictions under the International Traffic in Arms Regulations (ITAR) or the Export Administration Regulations (EAR). Export or transfer of this technical data and/or related information to any foreign person(s) or entity(ies), either within the U.S. or outside of the U.S., may require advance export authorization by the appropriate U.S. Government agency prior to export or transfer. In addition, technical data may not be exported or transferred to certain countries or specified designated nationals identified by U.S. embargo controls without prior export authorization. By accepting this email and any attachments, all recipients confirm that they understand and will comply with all applicable ITAR, EAR and embargo compliance requirements.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100921/b27f9ac1/attachment-0001.htm
More information about the syslog-ng
mailing list