On Sun, 2011-03-06 at 11:01 -0600, Martin Holste wrote:
Of course, this makes perfect sense if you're an administrator or you're trying to analyze logs from already-written applications, but what if you're the one who writes these applications and you'd like to make them easier to analyze later on?
You sir, are on the right track. When I do web application assessments, the biggest thing I recommend after getting the bugs fixed is to make sure that the business logic can be properly audited and alerted on if necessary.
The idea I've been toying with is defining some kind of a standardized commenting scheme for logging-related calls in source code. Using this, the code could be parsed automatically and a list of log messages could easily be generated that could be used later for documentation or even generating a pattern database.
This is a worthy idea! I think the big problem is that you are asking normal developers to understand how their logs are going to be parsed, so every developer would have to learn the pattern-db format to produce these comments. While laudable, I do not think this is realistic. It would be much more successful if the developers did not have to care what the format would be and only had to specify exactly what fields would be sent. This isn't possible, though, because at this stage, we want the messages to be both human readable and machine parsable at the same time. That requires the developers who are used to writing human-readable messages to know how to parse them, or it would require them sending only machine-parsable messages. To me, spitting out JSON messages accomplishes both, e.g.: { "class":"secevt", "tags": [ "ssh", "system" ], "fields": { "verdict": "accepted" } } I think that is both human readable and is obviously machine parsable. So, I think the real answer is to convince developers to begin logging in a format that can be parsed. Even WELF or CSV would be a major step forward. Otherwise, we're left with what we have now, which is to try to use patternize to divine what the developers meant, because precious few developers would both bother to learn the pattern format and get it right.
Also, in order to be feasible the taxonomy (e.g. the naming of fields and tags) should be agreed upon, with all the developers out there. That's going to be tough. -- Bazsi