[syslog-ng] logic and duplicate suppression

[Martin Holste]

There are a number of high-level ways of handling this kind of task.
Here is my philosophy:

Disk is cheap.  Log everything and become efficient at
querying/grepping/reporting instead of pre-filtering.  This is
especially important for security because even the most mundane logs
can be critical later.   The way I handle your presented tasks is to
normalize incoming logs as much as possible with Syslog-NG and dump
them into SQL.  I can then run periodic queries against the SQL with
very fine-grained control for alerting, retention, or whatever
higher-level task you're looking to do.

So, for your example of handling ssh messages differently depending on
the source address, I have a SQL column for source address and then I
can do "WHERE INET_ATON(source_ip) NOT BETWEEN INET_ATON("x.x.x.x")
AND INET_ATON("y.y.y.y")" in my query.  For reporting, I can do "GROUP
BY INET_ATON(source_ip)-MOD(INET_ATON(source_ip), 256)" to group by a
class C subnet.

Maybe this is more than you want to do in your case, but it sounds to
me like maybe you're ready for some functionality beyond manually
reading through the log files.  There are plenty of ready-made log
collectors out there:  Balabit makes a nice solution in their Store
Box, Clayton has his Logzilla (php-syslog-ng) project, or if you're
under 500 MB per day of logs, I highly recommend the free Splunk
Personal Edition which is phenomenal.

My belief is that your time would be better spent setting up a solid
apparatus for querying and reporting than on trying to get Syslog-NG
to filter in the specific ways you want it to.

On Wed, Jul 28, 2010 at 10:04 AM, John Kristoff <jtk at cymru.com> wrote:
> I have a couple of scenarios where I'm looking to enhance how I handle
> and process some logs.  I'm looking for suggestions on what my options
> are, but maybe these are potential feature requests?
>
> 1. In using a parser (cvs or the patterndb), I'd like to use some
> conditionals based on a resultant macro value.  So for example, if I
> have an sshd authentication log message with a source address in a
> macro and that address is contained w/in a specific prefix, I'd like to
> handle that message differently.  Perhaps not log it all or set another
> MACRO to a certain value.
>
> 2. I'd like to be able to suppress duplicate messages even if they are
> not necessarily contiguous at the destination.  So for example, if I
> have a SSH client that generates a log of its SSH client protocol
> and software, I don't need to see that over and over again (e.g. as
> you might commonly see today in SSH brute force attacks).
>
> John
> ______________________________________________________________________________
> Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
> Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
> FAQ: http://www.campin.net/syslog-ng/faq.html
>
>