[syslog-ng] Windows Agent default install template

[Fekete Robert]

Hi Evan,

To make this work as it really should, I think we should take a slightly 
different angle at the problem:

- AFAIK, when using the RFC5424 message format, the agent adds every message 
field to the SDATA as name-value pairs (except for the EVENT_MESSAGE). So all 
this data is already available in a parsed and structured format. This means 
that only the EVENT_MESSAGE part needs to be identified and parsed.

- Since the Event ID is already available as a macro, it is easy to filter only 
the messages of the specific Event ID. Now we would only need two things: 
patterns for the EVENT_MESSAGE part of the messages (if needed), and a way to 
apply these patterns to these messages. Something like:
" if $EVENT_ID == x , then parse EVENT_MESSAGE with EVENT_ID_x.xml "
I _think_ that extending the patternb in syslog-ng to take templates so that it 
can be used only on the EVENT_MESSAGE part should be relatively simple, but I am 
not a programmer, so I might err here. Then only the problem of creating the 
appropriate configuration remains (adding tons of filters and patterndb parsers 
does not sound too convenient), but I guess this can be solved as well.

- There are two other things that syslog-ng could do: syslog-ng should recognize 
if it received a message that contains an eventlog message (this should be 
fairly simple, for example, by checking if the EVENT_ID sdata field is set), and 
act accordingly (for example, parse the message with a specific patterndb, or 
when writing the message to a file, use an eventlog-specific template to make it 
more readable, for example, the template that you suggested).

- Another useful thing could be to simplify all the above. Starting with 
syslog-ng Premium Edition 5 LTS (due to be released in a few weeks), the 
syslog-n Windows agent and the Premium Edition will use the same code-base, 
meaning that if there are patterns for the eventlog messages, then technically, 
it could be possible in the future add the pattern database to the Windows 
agent, and do the parsing on the clients. Then the agent could transfer the 
already parsed messages to the server, which then just had to use the parsed 
data to whatever you wanted to.

Kind Regards,

Robert

On 09/27/2013 05:21 PM, Evan Rempel wrote:
>
> We are in the process of a hug log filtering/profiling project where we are using syslog-ng to
> profile/classify *EVERY* message that comes through our syslog server.
> This means that we will have 1,000's of patterns in our patterndb, but we accept that.
>
> We have purchased the Syslog-ng Agent for Windows and are not profiling all of our Windows messages as well.
>
> Now to the challenge. The default message template is very difficult to parse using the pattern database.
> The problem is that many of the fields from a windows computer can contain spaces in them, and there
> are no delimiters in the default template. Additionally, the $EVENT_ID that windows places on
> each message, which is unique for each message type within the scope of each application that is logging, is placed
> at the end of the log line after the free form dext description. making a patterndb pattern to parse through the
> free form text just to get to the Event ID is very difficult.
>
>
> I wanted to open the discussion of using a different, and IMHO much more usefull, default template
> for MS Windows event logs. The pattern below has the following properties:
>
> - wraps each field in a [] pair
> - places the most definitive value, the Event ID, at the beginning of the template
> - places the least definitive and unstructured data, EVENT_MESSAGE, at the end of the template
>
>
> EventID[${EVENT_ID}] Log[${EVENT_NAME}] Type[${EVENT_TYPE}] User[${EVENT_USERNAME}] ${EVENT_MESSAGE}
>
>
> Let the discussion begin!
>
> Thanks for listening.
>