<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">Hi Evan,<br>
<br>
thank you for sharing your thoughts -- and I am really interested
in what others have to say on this. As Robert pointed out, when we
ourselves want to deal with structured event log information
coming from the Windows Agent we advise everyone to use the
RFC5424 format and use the SDATA fields for filtering or do
statistics based on the various fields in the event log messages.
The RFC3164-formatting of such messages are, as a result, more of
a fallback solution and we haven't thought much about its
parseability (at least I haven't, maybe Bazsi or the others will
correct me :)).<br>
<br>
Nevertheless, you are right that transforming structured
information into a format that is hard to parse is never a good
idea, and also that formatting event log messages better in the
Agent would allow us to use a pattern database on the other side.
It is true that it'd be "nicer" and probably more error-safe to
transfer the messages in a structured way (via RFC5424,
json-destination&source or some other more efficient protocol
we'd add) and match for name-value pairs in this structured data
within patterns (eg. "if EVENTID==X and EVENTTYPE==Y than
match()") -- and that'd happen over time. However, that will
require an overhaul of the current pattern database model which is
heavily focused on the $MSG part of the logs, while simply
agreeing on a more sensible default format for that $MSG part
would allow us to start writing and using patterns to parse and
classify messages from event logs instantly.<br>
<br>
So I am all for coming up with a more sensible default and if
there's an agreement I'm more than happy to include it (even as a
default) in the next version of the Agent.<br>
<br>
By the way, while writing your patterns, you might want to take a
look at this, if you haven't already:<br>
<br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<a
href="https://github.com/balabit/syslog-ng-patterndb/tree/master/os/windows2008">
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
</a><a
href="https://github.com/balabit/syslog-ng-patterndb/blob/master/os/windows2008/win2008.xml">https://github.com/balabit/syslog-ng-patterndb/blob/master/os/windows2008/win2008.xml</a><br>
<br>
This uses the current format, but tries to classify the most
important messages that appear on a Windows 2008 Server and
extract much more information than just the event ID or the
username -- you could use it as a basis. And of course we'd be
happy to include your patterns here as well if you are willing to
share them.<br>
<br>
Thanks,<br>
<br>
Peter<br>
(who is, for the record, responsible for the commercial side of
the syslog-ng product line at BalaBit)<br>
<br>
On 09/30/2013 05:28 PM, Evan Rempel wrote:<br>
</div>
<blockquote cite="mid:52499887.9010708@uvic.ca" type="cite">
<pre wrap="">Yes, I know that this can all be done with RFC5424 message transport, but
the Syslog-ng Agent for Windows has a message template for RFC3164 message transport
and that is what I am referring to.
When using RFC3164 it is even more important that the message have a format/template
that can be dealt with by a central log collection destination. For syslog-ng that
means that the message should be easy to parse with patterndb tools.
In my opinion the default message template makes parsing more difficult than it should
because it takes the structured data (MESSAGE_ID, EVENT_NAME, EVENT_MESSAGE etc) and
concatenates them together without any structure. Starting with structured data
and changing it to unstructured data is not helpful. I realize that having only one
message body in RFC3164 is difficult, but using a template like the one I have
proposed minimizes the difficulty.
Thanks for your thorough response.
Evan.
On 09/30/2013 01:05 AM, Fekete Robert wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi Evan,
To make this work as it really should, I think we should take a slightly
different angle at the problem:
- AFAIK, when using the RFC5424 message format, the agent adds every message
field to the SDATA as name-value pairs (except for the EVENT_MESSAGE). So all
this data is already available in a parsed and structured format. This means
that only the EVENT_MESSAGE part needs to be identified and parsed.
- Since the Event ID is already available as a macro, it is easy to filter only
the messages of the specific Event ID. Now we would only need two things:
patterns for the EVENT_MESSAGE part of the messages (if needed), and a way to
apply these patterns to these messages. Something like:
" if $EVENT_ID == x , then parse EVENT_MESSAGE with EVENT_ID_x.xml"
I _think_ that extending the patternb in syslog-ng to take templates so that it
can be used only on the EVENT_MESSAGE part should be relatively simple, but I am
not a programmer, so I might err here. Then only the problem of creating the
appropriate configuration remains (adding tons of filters and patterndb parsers
does not sound too convenient), but I guess this can be solved as well.
- There are two other things that syslog-ng could do: syslog-ng should recognize
if it received a message that contains an eventlog message (this should be
fairly simple, for example, by checking if the EVENT_ID sdata field is set), and
act accordingly (for example, parse the message with a specific patterndb, or
when writing the message to a file, use an eventlog-specific template to make it
more readable, for example, the template that you suggested).
- Another useful thing could be to simplify all the above. Starting with
syslog-ng Premium Edition 5 LTS (due to be released in a few weeks), the
syslog-n Windows agent and the Premium Edition will use the same code-base,
meaning that if there are patterns for the eventlog messages, then technically,
it could be possible in the future add the pattern database to the Windows
agent, and do the parsing on the clients. Then the agent could transfer the
already parsed messages to the server, which then just had to use the parsed
data to whatever you wanted to.
Kind Regards,
Robert
On 09/27/2013 05:21 PM, Evan Rempel wrote:
</pre>
<blockquote type="cite">
<pre wrap="">
We are in the process of a hug log filtering/profiling project where we are using syslog-ng to
profile/classify *EVERY* message that comes through our syslog server.
This means that we will have 1,000's of patterns in our patterndb, but we accept that.
We have purchased the Syslog-ng Agent for Windows and are not profiling all of our Windows messages as well.
Now to the challenge. The default message template is very difficult to parse using the pattern database.
The problem is that many of the fields from a windows computer can contain spaces in them, and there
are no delimiters in the default template. Additionally, the $EVENT_ID that windows places on
each message, which is unique for each message type within the scope of each application that is logging, is placed
at the end of the log line after the free form dext description. making a patterndb pattern to parse through the
free form text just to get to the Event ID is very difficult.
I wanted to open the discussion of using a different, and IMHO much more usefull, default template
for MS Windows event logs. The pattern below has the following properties:
- wraps each field in a [] pair
- places the most definitive value, the Event ID, at the beginning of the template
- places the least definitive and unstructured data, EVENT_MESSAGE, at the end of the template
EventID[${EVENT_ID}] Log[${EVENT_NAME}] Type[${EVENT_TYPE}] User[${EVENT_USERNAME}] ${EVENT_MESSAGE}
Let the discussion begin!
Thanks for listening.
</pre>
</blockquote>
<pre wrap="">
______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<pre wrap="">
______________________________________________________________________________
Member info: <a class="moz-txt-link-freetext" href="https://lists.balabit.hu/mailman/listinfo/syslog-ng">https://lists.balabit.hu/mailman/listinfo/syslog-ng</a>
Documentation: <a class="moz-txt-link-freetext" href="http://www.balabit.com/support/documentation/?product=syslog-ng">http://www.balabit.com/support/documentation/?product=syslog-ng</a>
FAQ: <a class="moz-txt-link-freetext" href="http://www.balabit.com/wiki/syslog-ng-faq">http://www.balabit.com/wiki/syslog-ng-faq</a>
</pre>
</blockquote>
<br>
</body>
</html>