Hello, The attached patch comes from http://dev.riseup.net/patches/syslog-ng what it does is provide a simple filter to strip out unwanted regular expressions from logs, as well as an IP alias that enables you to strip out IP addresses from your logs. This patch has been applied to the Debian package of syslog-ng, I am writing here to let people know about it and submit it for consideration into syslog-ng.
From the README:
This patch adds the capability to syslog-ng that allows you to strip out any given regexp or all IP addresses from log messages before they are written to disk. The goal is to give the system administrator the means to implement site logging policies, by allowing them easy control over exactly what data they retain in their logfiles, regardless of what a particular daemon might think is best. Background: Data retention has become a hot legal topic for ISPs and other Online Service Providers (OSPs). There are many instances where it is preferable to keep less information on users than is collected by default on many systems. In the United States it is not currently required to retain data on users of a server, but you may be required to provide all data on a user which you have retained. OSPs can protect themselves from legal hassles and added work by choosing what data they wish to retain.
From "Best Practices for Online Service Providers" (http://www.eff.org/osp):
As an intermediary, the OSP finds itself in a position to collect and store detailed information about its users and their online activities that may be of great interest to third parties. The USA PATRIOT Act also provides the government with expanded powers to request this information. As a result, OSP owners must deal with requests from law enforcement and lawyers to hand over private user information and logs. Yet, compliance with these demands takes away from an OSP's goal of providing users with reliable, secure network services. In this paper, EFF offers some suggestions, both legal and technical, for best practices that balance the needs of OSPs and their users' privacy and civil liberties. Rather than scrubbing the information you don't want in logs, this patch ensures that the information is never written to disk. Also, for those daemons which log through syslog facilities, this patch provides a convenient single configuration to limit what you wish to log. Here are some related links: Best Practices for Online Service Providers http://www.eff.org/osp http://www.eff.org/osp/20040819_OSPBestPractices.pdf EPIC International Data Retention Page http://www.epic.org/privacy/intl/data_retention.html Working Paper on Usage Log Data Management (from Computer, Freedom, and Privacy conference) http://cryptome.org/usage-logs.htm How to use it This patch adds the filter "strip". For example: filter f_strip {strip(<regexp>);}; This will strip out all matches of the regular expression on logs to which the filter is applied and replaces all matches with the fixed length four dashes ("----"). In place of a regular expression, you can put "ips", which will replace all internet addresses with 0.0.0.0 <http://0.0.0.0>. For example: filter f_strip {strip(ips);}; You can alter what the replacement strings are by using replace: replace(ips,"0.0.0.0 <http://0.0.0.0>") <--- this is the same as strip(ips) replace(<regex>,"----") <--- this is the same as strip(regex) We provide a debian package of 1.6.7 with this patch added (the repository is http://deb.riseup.net/debian unstable main), or you can retrieve the patch yourself from http://dev.riseup.net/websvn/listing.php?repname=syslog-ng-anon&path=%2F&sc=... apply it with: # patch -p1 < syslog-ng-anon.diff