<br><div class="gmail_quote">On Wed, Jun 23, 2010 at 11:34 PM, Hendrik Pahl <span dir="ltr"><<a href="mailto:pahl@team-datentechnik.de">pahl@team-datentechnik.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi,<br>
<br>
...<br>
> That said, it does not soundlike you need to use it for what you're<br>
> trying to do.<br>
<br>
Okay, i already had the feeling patterndb was not the one really<br>
giving me a solution. I simply need something to bring down the<br>
relevant loglines, since 1.5M lines/month in a logfile/different<br>
logfiles are simply much to much to monitor/read.<br>
<br>
Grepping after "error" or "warning" or "failure" is just one approach,<br>
but never will be the only one, since this might kick out things i<br>
wanna definitely see.<br>
<br>
currently i'm looking at logfiles and size down the amount of lines by<br>
piping the cat output into sed, which kicks out the informational and<br>
overhead lines. this ia an iterative apporach, since i refine the sed<br>
expression time to time.<br>
<br>
How are others managing this issue?<br>
<div><div></div><div class="h5"><br><br></div></div></blockquote><div><br></div><div>Hi,</div><div><br></div><div>So by the sounds of it, you're looking to match certain lines based on regex, and then you want to read the surrounding lines of your match. Is this accurate? This is *always* going to be tedious without the help of some software. Syslog-ng will get you to the point where you can match whatever you want in the log line contents and then write that out to a file, ignoring everything that doesn't get matched. You can do this with either filters{} or patterndb. </div>
<div><br></div><div>If you need alerts based on matching, syslog-ng can do this do if you output matches to alerting scripts; however, a log file analyzer may be better suited. Simple Event Correlator (SEC) is a popular tool, but there are plenty of log security tools that you can use that are easy to write matches for. OSSEC is a very good tool which is easy to configure, it's just noisy until you customize it. </div>
<div><br></div><div>As for your problem where you are wanting to see the log lines above and below the line(s) you are grepping/regexing for, you're going to have to look into flexible log viewing applications such as Splunk, or if your log volume isn't insane, some sort of syslog frontend like LogZilla. The problem with Splunk though? Money. Splunk is so expensive because it's the only thing out there that does everything log related well - lots of people have the same problem as you except in some cases need searchable access to gigabytes or terabytes worth of logs. You can look for free alternatives, but they will all be using an sql-based database to store logs and you will most likely end up finding too many tools that are trying to be security event alerting packages rather than log viewing apps.</div>
<div><br></div><div>You should also consider why you are needing to parse through so many logs manually, what is generating that volume, and whether you can make the log messages the application outputs more relevant. If things are constantly needing your attention to the extent that you cannot make enough matching rules to alert on, or the error-severity messages aren't including the actual problem which is forcing you to read what happened before it, then they are broken to begin with and should be fixed at the application.</div>
</div><div><br></div>Regards,<br clear="all"><br>-- <br>Lance Laursen<br>Demonware Systems Engineer<br>