There are a number of high-level ways of handling this kind of task. Here is my philosophy: Disk is cheap. Log everything and become efficient at querying/grepping/reporting instead of pre-filtering. This is especially important for security because even the most mundane logs can be critical later. The way I handle your presented tasks is to normalize incoming logs as much as possible with Syslog-NG and dump them into SQL. I can then run periodic queries against the SQL with very fine-grained control for alerting, retention, or whatever higher-level task you're looking to do. So, for your example of handling ssh messages differently depending on the source address, I have a SQL column for source address and then I can do "WHERE INET_ATON(source_ip) NOT BETWEEN INET_ATON("x.x.x.x") AND INET_ATON("y.y.y.y")" in my query. For reporting, I can do "GROUP BY INET_ATON(source_ip)-MOD(INET_ATON(source_ip), 256)" to group by a class C subnet. Maybe this is more than you want to do in your case, but it sounds to me like maybe you're ready for some functionality beyond manually reading through the log files. There are plenty of ready-made log collectors out there: Balabit makes a nice solution in their Store Box, Clayton has his Logzilla (php-syslog-ng) project, or if you're under 500 MB per day of logs, I highly recommend the free Splunk Personal Edition which is phenomenal. My belief is that your time would be better spent setting up a solid apparatus for querying and reporting than on trying to get Syslog-NG to filter in the specific ways you want it to. On Wed, Jul 28, 2010 at 10:04 AM, John Kristoff <jtk@cymru.com> wrote:
I have a couple of scenarios where I'm looking to enhance how I handle and process some logs. I'm looking for suggestions on what my options are, but maybe these are potential feature requests?
1. In using a parser (cvs or the patterndb), I'd like to use some conditionals based on a resultant macro value. So for example, if I have an sshd authentication log message with a source address in a macro and that address is contained w/in a specific prefix, I'd like to handle that message differently. Perhaps not log it all or set another MACRO to a certain value.
2. I'd like to be able to suppress duplicate messages even if they are not necessarily contiguous at the destination. So for example, if I have a SSH client that generates a log of its SSH client protocol and software, I don't need to see that over and over again (e.g. as you might commonly see today in SSH brute force attacks).
John ______________________________________________________________________________ Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng FAQ: http://www.campin.net/syslog-ng/faq.html