[syslog-ng] few questions about patterndb

[Lance Laursen]

On Wed, Jun 23, 2010 at 11:34 PM, Hendrik Pahl <pahl at team-datentechnik.de>wrote:

> Hi,
>
> ...
> > That said, it does not soundlike you need to use it for what you're
> > trying to do.
>
> Okay, i already had the feeling patterndb was not the one really
> giving me a solution. I simply need something to bring down the
> relevant loglines, since 1.5M lines/month in a logfile/different
> logfiles are simply much to much to monitor/read.
>
> Grepping after "error" or "warning" or "failure" is just one approach,
> but never will be the only one, since this might kick out things i
> wanna definitely see.
>
> currently i'm looking at logfiles and size down the amount of lines by
> piping the cat output into sed, which kicks out the informational and
> overhead lines. this ia an iterative apporach, since i refine the sed
> expression time to time.
>
> How are others managing this issue?
>
>
>
Hi,

So by the sounds of it, you're looking to match certain lines based on
regex, and then you want to read the surrounding lines of your match. Is
this accurate? This is *always* going to be tedious without the help of some
software. Syslog-ng will get you to the point where you can match whatever
you want in the log line contents and then write that out to a file,
ignoring everything that doesn't get matched. You can do this with either
filters{} or patterndb.

If you need alerts based on matching, syslog-ng can do this do if you output
matches to alerting scripts; however, a log file analyzer may be better
suited. Simple Event Correlator (SEC) is a popular tool, but there are
plenty of log security tools that you can use that are easy to write matches
for. OSSEC is a very good tool which is easy to configure, it's just noisy
until you customize it.

As for your problem where you are wanting to see the log lines above and
below the line(s) you are grepping/regexing for, you're going to have to
look into flexible log viewing applications such as Splunk, or if your log
volume isn't insane, some sort of syslog frontend like LogZilla. The problem
with Splunk though? Money. Splunk is so expensive because it's the only
thing out there that does everything log related well - lots of people have
the same problem as you except in some cases need searchable access to
gigabytes or terabytes worth of logs. You can look for free alternatives,
but they will all be using an sql-based database to store logs and you will
most likely end up finding too many tools that are trying to be security
event alerting packages rather than log viewing apps.

You should also consider why you are needing to parse through so many logs
manually, what is generating that volume, and whether you can make the log
messages the application outputs more relevant. If things are constantly
needing your attention to the extent that you cannot make enough matching
rules to alert on, or the error-severity messages aren't including the
actual problem which is forcing you to read what happened before it, then
they are broken to begin with and should be fixed at the application.

Regards,

-- 
Lance Laursen
Demonware Systems Engineer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.balabit.hu/pipermail/syslog-ng/attachments/20100624/2b2faaba/attachment.htm